[
https://issues.apache.org/jira/browse/IMPALA-9461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltán Borók-Nagy updated IMPALA-9461:
--------------------------------------
Description:
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard
metadata store didn't exist:
{noformat}
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
auto-creation is turned off{noformat}
In impalad.ERROR:
{noformat}
hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0,
kerbTicketCachePath=(NULL), userName=(NULL)) error:
ResourceNotFoundException: Requested resource not found: Table:
impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID:
6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.
FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist in
region us-west-2; auto-creation is turned off
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
{noformat}
I suspected that the S3Guard metadata store initialization failed, so I checked
load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
{noformat}
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store:
dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
billing-mode=per-request
description=S3Guard metadata store in DynamoDB
name=impala-test-uswest2-1
persist.authoritative.bit=true
read-capacity=0
region=us-west-2
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
size=9791493
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName:
impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH},
{AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,
CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput:
{NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits:
0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn:
arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId:
c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode:
PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST
2020},}
write-capacity=0
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store:
dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.
20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning
DynamoDB Store
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store:
duration 1:38.538s
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981
items in batches of 25
{noformat}
So based on this impala-test-uswest2-1 is successfully created in the region
us-west-2, but later it became unreachable/deleted.
was:
I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard
metadata store didn't exist:
{noformat}
I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
auto-creation is turned off{noformat}
In impalad.ERROR:
{noformat}
hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0,
kerbTicketCachePath=(NULL), userName=(NULL)) error:
ResourceNotFoundException: Requested resource not found: Table:
impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400;
Error Code: ResourceNotFoundException; Request ID:
6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.
FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist
in region us-west-2; auto-creation is turned off
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
{noformat}
I suspected that the S3Guard metadata store initialization failed, so I checked
load-test-warehouse-snapshot.log, but seems like S3Guard init was successful:
{noformat}
20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store:
dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.
Metadata Store Diagnostics:
ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
billing-mode=per-request
description=S3Guard metadata store in DynamoDB
name=impala-test-uswest2-1
persist.authoritative.bit=true
read-capacity=0
region=us-west-2
retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100 MILLISECONDS)
size=9791493
status=ACTIVE
table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName:
impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH},
{AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,
CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput:
{NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits:
0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn:
arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId:
c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode:
PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST
2020},}
write-capacity=0
20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store:
dynamodb://impala-test-uswest2-1 scheme: dynamodb
20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store
DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
is initialized.
20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning
DynamoDB Store
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB Store:
duration 1:38.538s
20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981
items in batches of 25
{noformat}
So based on this impala-test-uswest2-1 is successfully created in the region
us-west-2, but later it became unreachable/deleted.
> S3Guard metadata store cannot be found
> --------------------------------------
>
> Key: IMPALA-9461
> URL: https://issues.apache.org/jira/browse/IMPALA-9461
> Project: IMPALA
> Issue Type: Bug
> Reporter: Zoltán Borók-Nagy
> Priority: Major
> Labels: broken-build
>
> I've seen this in a build on S3 + S3Guard. Many tests failed because S3Guard
> metadata store didn't exist:
> {noformat}
> I0304 00:34:48.754648 17557 status.cc:126] couldn't retrieve FileSystem:
> DynamoDB table 'impala-test-uswest2-1' does not exist in region us-west-2;
> auto-creation is turned off{noformat}
> In impalad.ERROR:
> {noformat}
> hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0,
> kerbTicketCachePath=(NULL), userName=(NULL)) error:
> ResourceNotFoundException: Requested resource not found: Table:
> impala-test-uswest2-1 not found (Service: AmazonDynamoDBv2; Status Code: 400;
> Error Code: ResourceNotFoundException; Request ID:
> 6A5PINL6IDFF93BNDSSOQCG6JFVV4KQNSO5AEMVJF66Q9ASUAAJG)java.io.
> FileNotFoundException: DynamoDB table 'impala-test-uswest2-1' does not exist
> in region us-west-2; auto-creation is turned off
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStoreTableManager.initTable(DynamoDBMetadataStoreTableManager.java:216)
> at
> org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:427)
> at
> org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:109)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:418)
> at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3320)
> {noformat}
> I suspected that the S3Guard metadata store initialization failed, so I
> checked load-test-warehouse-snapshot.log, but seems like S3Guard init was
> successful:
> {noformat}
> 20/03/03 22:03:36 INFO s3guard.S3GuardTool: Create metadata store:
> dynamodb://impala-test-uswest2-1 scheme: dynamodb
> 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Metadata store
> DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
> tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
> is initialized.
> Metadata Store Diagnostics:
> ARN=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1
> billing-mode=per-request
> description=S3Guard metadata store in DynamoDB
> name=impala-test-uswest2-1
> persist.authoritative.bit=true
> read-capacity=0
> region=us-west-2
> retryPolicy=ExponentialBackoffRetry(maxRetries=9, sleepTime=100
> MILLISECONDS)
> size=9791493
> status=ACTIVE
> table={AttributeDefinitions: [{AttributeName: child,AttributeType: S},
> {AttributeName: parent,AttributeType: S}],TableName:
> impala-test-uswest2-1,KeySchema: [{AttributeName: parent,KeyType: HASH},
> {AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,
> CreationDateTime: Mon Mar 02 22:02:44 PST 2020,ProvisionedThroughput:
> {NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits:
> 0},TableSizeBytes: 9791493,ItemCount: 47304,TableArn:
> arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1,TableId:
> c0fad823-9fb7-48ba-b803-661a2b4052ee,BillingModeSummary: {BillingMode:
> PAY_PER_REQUEST,LastUpdateToPayPerRequestDateTime: Mon Mar 02 22:02:44 PST
> 2020},}
> write-capacity=0
> 20/03/03 22:03:38 INFO s3guard.S3GuardTool: Create metadata store:
> dynamodb://impala-test-uswest2-1 scheme: dynamodb
> 20/03/03 22:03:40 INFO s3guard.S3GuardTool: Metadata store
> DynamoDBMetadataStore{region=us-west-2, tableName=impala-test-uswest2-1,
> tableArn=arn:aws:dynamodb:us-west-2:360379543683:table/impala-test-uswest2-1}
> is initialized.
> 20/03/03 22:03:40 INFO s3guard.DynamoDBMetadataStore: Starting: Pruning
> DynamoDB Store
> 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Pruning DynamoDB
> Store: duration 1:38.538s
> 20/03/03 22:05:18 INFO s3guard.DynamoDBMetadataStore: Finished pruning 25981
> items in batches of 25
> {noformat}
> So based on this impala-test-uswest2-1 is successfully created in the region
> us-west-2, but later it became unreachable/deleted.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
