[
https://issues.apache.org/jira/browse/HADOOP-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15943084#comment-15943084
]
Steve Loughran commented on HADOOP-14239:
-----------------------------------------
I'm going to mark this up as a dupe of HADOOP-11572, which is where I first sat
down to look at this problem, or at least the related one of concurrent object
delete failure.
I stopped there as I concluded I didn't know enough about how things fail. It
may be someone else deleted the object —race condition, that being the
likeliest. Or its permission related, auth related, etc etc. Without knowing
all the failure modes, I wasn't confident I could implement the right policy.
If you have more insight there, that'd be good. As usual, the tests are as
important as the production code; the landsat-pds repo the read only one to
look at.
> S3A Retry Multiple S3 Key Deletion
> ----------------------------------
>
> Key: HADOOP-14239
> URL: https://issues.apache.org/jira/browse/HADOOP-14239
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: EC2, AWS
> Reporter: Kazuyuki Tanimura
>
> When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3
> keys at once.
> Although this is a great feature, it becomes problematic when AWS fails
> deleting some S3 keys out of the deletion list. The aws-java-sdk internally
> retries to delete them, but it does not help because it simply retries the
> same list of S3 keys including the successfully deleted ones. In that case,
> all successive retries fail deleting previously deleted keys since they do
> not exist any more. Eventually it throws an Exception and leads to a job
> failure entirely.
> Luckily, the AWS API reports which keys it failed to delete. We should retry
> only for the keys that failed to be deleted from S3A
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
