[
https://issues.apache.org/jira/browse/HADOOP-16280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran resolved HADOOP-16280.
-------------------------------------
Fix Version/s: 3.3.0
Resolution: Duplicate
HADOOP-16490 added exactly this retry logic
> S3Guard: Retry failed read with backoff in Authoritative mode when file can
> be opened
> -------------------------------------------------------------------------------------
>
> Key: HADOOP-16280
> URL: https://issues.apache.org/jira/browse/HADOOP-16280
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Reporter: Gabor Bota
> Priority: Major
> Fix For: 3.3.0
>
>
> When using S3Guard in authoritative mode a file can be reported from AWS S3
> that's missing like it is described in the following exception:
> {noformat}
> java.io.FileNotFoundException: re-open
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0
> at 0 on
> s3a://cloudera-dev-gabor-ireland/test/TMCDOR-021df1ad-633f-47b8-97f5-6cd93f0b82d0:
>
> com.amazonaws.services.s3.model.AmazonS3Exception: The specified key does not
> exist. (Service: Amazon S3; Status Code: 404; Error
> Code: NoSuchKey; Request ID: E1FF9EA9B5DBBD7E; S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=),
> S3 Extended Request ID:
> NzNIL4+dyA89WTnfbcwuYQK+hCfx51TfavwgC3oEvQI0IQ9M/zAspbXOfBIis8/nTolc4tRB9ik=:NoSuchKey
> {noformat}
> But the metadata in S3Guard (e.g dynamo db) is there, so it can be opened.
> The operation will not fail when it's opened, it will fail when we try to
> read it, so the call
> {noformat}
> FSDataInputStream is = guardedFs.open(testFilePath);{noformat}
> won't fail, but the next call
> {noformat}
> byte[] firstRead = new byte[text.length()];
> is.read(firstRead, 0, firstRead.length);
> {noformat}
> will fail with the exception message like what's above.
> Once Authoritative mode is on, we assume that there is no out of band
> operation, so the file will appear eventually. We should re-try in this case.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
