[
https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623561#action_12623561
]
Amareshwari Sriramadasu commented on HADOOP-3150:
-------------------------------------------------
There could be a race condition in finding out the final reducer.
For example, the last but one reducer reported as it is COMMIT_PENDING. It will
get a CommitTaskAction from Jobtracker. Before the last but one reducer reports
as SUCCEEDED, the last reducer goes to COMMIT_PENDING. Since JT notices there
is another unsuccessful reducer, the last reducer gets a commit task action,
but final attempt flag is not set. So, none of the attempts is sent a commit
task action with cleanup Job flag set.
Another scenario is, all the reducers are in COMMIT_PENDING at the same time.
Then none of them is considered to be final attempt.
Solutions:
1. Have a process launched by JT to do the cleanup at the Job completion. The
process should be aware of JobConf to call the code :
conf.getOutputCommiter().cleanupJob(JobContext)
2. Launch a standalone task at Job completion to do the cleanup Job.
Thoughts?
> Move task file promotion into the task
> --------------------------------------
>
> Key: HADOOP-3150
> URL: https://issues.apache.org/jira/browse/HADOOP-3150
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: 3150.patch, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt
>
>
> We need to move the task file promotion from the JobTracker to the Task and
> move it down into the output format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
