[
https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-15003:
------------------------------------
Attachment: HADOOP-13786-048.patch
Yetus didn't run the last patch, so lets try again.
Changes here minor; apart from staging committer opt out of uploads, & test
tuning, this is about documentation and marking up the DDB code for its retry
policy. Looks to me like we need to implement exception translation there as
well as retrying ops where appropriate.
* staging committers can opt out of aborting all uploads in job cleanup;
consistent with the original netflix committer. This allows >1 partitioned
committer to write to the same directory tree as long as you are confident that
they really are writing to different partitions (race conditions in parallel
job commits, see)
* Add SocketTimeoutException as retryable on idempotent calls
* add Retries policy to DynamoDBMetadataStore. This highlights there's little
exception translation taking place, and not enough retries. Not added any more
retry/translation; that's a S3Guard issue. Just documented what's happening to
see what is needed and filed HADOOP-15035.
* DynamoDB uses same idempotency policy on delete as S3 (i.e. use the same
constant in retry(delete))
* update committer documentation on configuration, more troubleshooting, whole
section in index.md on retries and recovery
* Tests: ITestS3AHugeMagicCommits checks pending upload count before (==1) and
after (==0) commit. Didn't find any problems, as expected.
> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>
> Key: HADOOP-15003
> URL: https://issues.apache.org/jira/browse/HADOOP-15003
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch,
> HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch,
> HADOOP-13786-046.patch, HADOOP-13786-047.patch, HADOOP-13786-048.patch
>
>
> This is a Yetus only JIRA created to have Yetus review the
> HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR
> [https://github.com/apache/hadoop/pull/282] is stopping this happening in
> HADOOP-14971.
> Reviews should go into the PR/other task
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
