[
https://issues.apache.org/jira/browse/HADOOP-16349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856648#comment-16856648
]
Steve Loughran commented on HADOOP-16349:
-----------------------------------------
log of the attempt. Note how long it takes
{code}
2019-06-05 13:06:52,254 [main] DEBUG s3guard.DynamoDBClientFactory
(DynamoDBClientFactory.java:createDynamoDBClient(86)) - Creating DynamoDB
client in region eu-west-1
2019-06-05 13:06:52,372 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:initTable(1227)) - Binding to table
hwdev-steve-ireland-new
2019-06-05 13:06:52,736 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:initTable(1229)) - Table state:
{AttributeDefinitions: [{AttributeName: child,AttributeType: S},
{AttributeName: parent,AttributeType: S}],TableName:
hwdev-steve-ireland-new,KeySchema: [{AttributeName: parent,KeyType: HASH},
{AttributeName: child,KeyType: RANGE}],TableStatus: ACTIVE,CreationDateTime:
Thu Oct 11 18:51:14 BST 2018,ProvisionedThroughput: {LastIncreaseDateTime: Tue
Oct 30 16:48:45 GMT 2018,LastDecreaseDateTime: Tue Oct 30 18:00:03 GMT
2018,NumberOfDecreasesToday: 0,ReadCapacityUnits: 0,WriteCapacityUnits:
0},TableSizeBytes: 0,ItemCount: 0,TableArn:
arn:aws:dynamodb:eu-west-1:980678866538:table/hwdev-steve-ireland-new,TableId:
a7b0728a-f008-4260-b2a0-ff3dd03367d1,}
2019-06-05 13:06:52,774 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 334 ms
before next retry
2019-06-05 13:06:53,134 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 1349 ms
before next retry
2019-06-05 13:06:54,511 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 1242 ms
before next retry
2019-06-05 13:06:55,780 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 2883 ms
before next retry
2019-06-05 13:06:58,687 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 5926 ms
before next retry
2019-06-05 13:07:04,641 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 22395 ms
before next retry
2019-06-05 13:07:27,064 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 20600 ms
before next retry
2019-06-05 13:07:47,691 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 50479 ms
before next retry
2019-06-05 13:08:38,197 [main] DEBUG s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:getVersionMarkerItem(1318)) - Sleeping 115865 ms
before next retry
2019-06-05 13:10:34,328 [main] WARN s3guard.DynamoDBMetadataStore
(DynamoDBMetadataStore.java:verifyVersionCompatibility(1355)) - Table
hwdev-steve-ireland-new contains no version marker
2019-06-05 13:10:34,329 [main] ERROR s3guard.S3Guard
(S3Guard.java:getMetadataStore(113)) - Failed to instantiate metadata store
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore defined in
fs.s3a.metadatastore.impl: java.io.IOException: S3Guard table lacks version
marker. Table: hwdev-steve-ireland-new
java.io.IOException: S3Guard table lacks version marker. Table:
hwdev-steve-ireland-new
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.verifyVersionCompatibility(DynamoDBMetadataStore.java:1356)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initTable(DynamoDBMetadataStore.java:1256)
at
org.apache.hadoop.fs.s3a.s3guard.DynamoDBMetadataStore.initialize(DynamoDBMetadataStore.java:383)
at
org.apache.hadoop.fs.s3a.s3guard.S3Guard.getMetadataStore(S3Guard.java:102)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:383)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3338)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:136)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3387)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3355)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:497)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.hadoop.fs.shell.PathData.expandAsGlob(PathData.java:352)
at org.apache.hadoop.fs.shell.Command.expandArgument(Command.java:250)
at org.apache.hadoop.fs.shell.Command.expandArguments(Command.java:233)
at
org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:104)
at org.apache.hadoop.fs.shell.Command.run(Command.java:177)
at org.apache.hadoop.fs.FsShell.run(FsShell.java:327)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:390)
ls: S3Guard table lacks version marker. Table: hwdev-steve-ireland-new
{code}
> DynamoDBMetadataStore.getVersionMarkerItem() to log at info/warn on retry
> -------------------------------------------------------------------------
>
> Key: HADOOP-16349
> URL: https://issues.apache.org/jira/browse/HADOOP-16349
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.2.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
>
> If you delete the version marker from a S3Guard table, it appears to hang for
> 5 minutes.
> Only if you restart and turn logging to debug do you see that
> {{DynamoDBMetadataStore.getVersionMarkerItem()}} is sleeping and retrying.
> # log at warn
> # add entry to troubleshooting doc on the topic
> The cause of the failure can be any of
> * table being inited elsewhere: expectation, fast recovery
> * it's not a S3Guard table: it won't recover
> * it's a S3Guard table without a version marker: it won't recover.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
