Zoltán Borók-Nagy created IMPALA-9461:
-----------------------------------------
Summary: S3Guard metadata store cannot be found
Key: IMPALA-9461
URL: https://issues.apache.org/jira/browse/IMPALA-9461
Project: IMPALA
Issue Type: Bug
Reporter: Zoltán Borók-Nagy
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard
metadata store didn't exist:
{noformat}
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
auto-creation is turned off{noformat}
In impalad.ERROR:
{noformat}
ResourceNotFoundException: Requested resource not found: Table:
impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID:
C9PEDOBGV9AIT482FV3RMDEA4NVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.FileNotFoundException:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
auto-creation is turned offResourceNotFoundException: Requested resource not
found: Table: impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2;
Status Code: 400; Error Code: ResourceNotFoundException; Request ID:
C9PEDOBGV9AIT482FV3RMDEA4NVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.FileNotFoundException:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
auto-creation is turned off at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
at org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418){noformat}
I suspected that the S3Guard metadata store initialization failed, so I checked
load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
{noformat}
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store:
dynamodb://impala-test-uswest2-1 scheme: dynamodb20/03/03 22:03:36 INFO
s3guard.S3GuardTool: Create metadata store: dynamodb://impala-test-uswest2-1
scheme: dynamodb20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
billing-mode=per-request description=S3Guard metadata store in DynamoDB
name=impala-test-uswest2-1 persist.authoritative.bit=true read-capacity=0
region=us-west-2 retryPolicy=ExponentialBackoffRetry(maxRetries=9,
sleepTime=100 MILLISECONDS) size=9791493 status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName:
impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH},
{AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime:
Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput: {NumberOfDecreasesToday:
0,ReadCapacityUnits: 0,WriteCapacityUnits: 0},TableSizeBytes:
9791493,ItemCount: 47304,TableArn:
arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId:
c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode:
PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST
2020},} write-capacity=020/03/03 22:03:38 INFO s3guard.S3GuardTool: Create
metadata store: dynamodb://impala-test-uswest2-1 scheme: dynamodb20/03/03
22:03:40 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting:
Pruning DynamoDB Store20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore:
Pruning DynamoDB Store: duration 1:38.538s20/03/03 22:05:18 INFO
s3guard.DynamoDBMetadataStore: Finished pruning 25981 items in batches of
25{noformat}
So based on this impala-test-uswest2-1 is successfully created in the region
us-west-2, but later it became unreachable/deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
