[
https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16243916#comment-16243916
]
Steve Loughran commented on HADOOP-15003:
-----------------------------------------
Side issue, one for [~rdblue] in particular
If the job commit fails, should any existing {{_SUCCESS}} file be deleted?
Because things may be inconsistent & we shouldn't have any cue in the dir that
it contains valid data.
What I'd do is: delete the marker before any files are committed or deleted,
but after any conflict resolution checks which may fail the job have run. That
way, if a job commit is aborted: the existing marker is unchanged.
* directory committer: delete it for the APPEND case; retain for FAIL (its
deleted for free in OVERWRITE)
* partitioned committer: both OVERWRITE and APPEND to delete the marker, though
append will
Moot for the magic commit as it doesn't do any in-situ overwrites/appends of
data.
If you think we should do this, I can add it as a folowup JIRA; would need test
modification &c
> Merge S3A committers into trunk: Yetus patch checker
> ----------------------------------------------------
>
> Key: HADOOP-15003
> URL: https://issues.apache.org/jira/browse/HADOOP-15003
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Attachments: HADOOP-13786-041.patch, HADOOP-13786-042.patch,
> HADOOP-13786-043.patch, HADOOP-13786-044.patch, HADOOP-13786-045.patch,
> HADOOP-13786-046.patch
>
>
> This is a Yetus only JIRA created to have Yetus review the
> HADOOP-13786/HADOOP-14971 patch as a .patch file, as the review PR
> [https://github.com/apache/hadoop/pull/282] is stopping this happening in
> HADOOP-14971.
> Reviews should go into the PR/other task
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
