[
https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16225666#comment-16225666
]
Aaron Fabbri commented on HADOOP-14971:
---------------------------------------
Thanks for the notes [[email protected]]. Looks like you got the major
points. We also discussed:
* Compatibility. Looks like existing OutputCommitter APIs and configurations
are still supported. We didn't think any breaks in compatibility are
introduced by the patch.
* Retries: The policy is all in one place in the {{S3ARetryPolicy}}
constructor. Review of current implementation: looks good for now.
* "idempotent" delete: There doesn't seem to be an obvious best answer here.
Current patch does not make the flag {{DELETE_CONSIDERED_IDEMPOTENT}}
configurable; it is hard-coded to {{true}}. While delete is not idempotent,
treating it like it is for retries makes sense given that S3 is not strongly
consistent, and thus multi-client delete/modify scenarios are already racy.
This allows us to retry on transient S3 service issues and get more robust job
completion under throttling, etc.. I feel like making the idempotent flag
configurable would be safest, but I understand the counterpoint of "too much
configurability". I'm going to defer to [[email protected]] on this, so I'm
+1 on this part of the current patch.
I think this stuff is ready to go in once we (1) restore separate SDK/S3A retry
config knobs (2) finish any testing we want to do.
> Merge S3A committers into trunk
> -------------------------------
>
> Key: HADOOP-14971
> URL: https://issues.apache.org/jira/browse/HADOOP-14971
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
>
> Merge the HADOOP-13786 committer into trunk. This branch is being set up as a
> github PR for review there & to keep it out the mailboxes of the watchers on
> the main JIRA
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
