[
https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244157#comment-16244157
]
Steve Loughran commented on HADOOP-15003:
-----------------------------------------
Think I've found one little issue which could create problems, though not by
its own the root cause. Seem closer to replicating it though, and yes, seems
related to cleanup of jobs.
One little feature is that the committers were being (over?) zealous in
aborting all MPUs under their destination path, on the basis that failed tasks
could have left outstanding MPUs which, if the data were not persisted, would
not be enumerable by looking for .pendingset files.
But {{S3aFileSystem.listMultipartUploads(prefix)}} actually turns out to list
everything matching the prefix, even in parallel dirs, as it isn't adding a "/"
suffix to say "directory only". Which meant if you had a job commit to
"test/myjob", i'd also delete pending uploads to "test/myjob2"
There's more to it than that; I think I'll need to review all listing stuff to
be sure, but now I have some tests failing. Provided the tests themselves are
correct, then I'll be able to find/fix it.
> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>
> Key: HADOOP-15003
> URL: https://issues.apache.org/jira/browse/HADOOP-15003
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch,
> HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch,
> HADOOP-13786-046.patch
>
>
> This is a Yetus only JIRA created to have Yetus review the
> HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR
> [https://github.com/apache/hadoop/pull/282] is stopping this happening in
> HADOOP-14971.
> Reviews should go into the PR/other task
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
