[
https://issues.apache.org/jira/browse/HADOOP-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15951820#comment-15951820
]
Kazuyuki Tanimura commented on HADOOP-14239:
--------------------------------------------
I went over HADOOP-11572 and saw what you worked on.
I see the cases that partial transient deletion errors happen other than race
condition, permission, auth cases. Particularly when a Hadoop job creates
thousands of output part files, only a few files (like part-00003) fails out of
part-00000 to part-0xxxx. The majority of the files are deleted without issues,
there shouldn't be reasons to target particular part files that cannot be
deleted.
aws-java-sdk is populating the partial failure file list here
https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/AmazonS3Client.java#L2097
These are the insights that I have. If the above case makes you more confident
that the retrying on failed S3 keys is the right strategy, I will start adding
tests.
> S3A Retry Multiple S3 Key Deletion
> ----------------------------------
>
> Key: HADOOP-14239
> URL: https://issues.apache.org/jira/browse/HADOOP-14239
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
> Affects Versions: 2.8.0
> Environment: EC2, AWS
> Reporter: Kazuyuki Tanimura
>
> When fs.s3a.multiobjectdelete.enable == true, It tries to delete multiple S3
> keys at once.
> Although this is a great feature, it becomes problematic when AWS fails
> deleting some S3 keys out of the deletion list. The aws-java-sdk internally
> retries to delete them, but it does not help because it simply retries the
> same list of S3 keys including the successfully deleted ones. In that case,
> all successive retries fail deleting previously deleted keys since they do
> not exist any more. Eventually it throws an Exception and leads to a job
> failure entirely.
> Luckily, the AWS API reports which keys it failed to delete. We should retry
> only for the keys that failed to be deleted from S3A
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
