[
https://issues.apache.org/jira/browse/HADOOP-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12628695#action_12628695
]
Matei Zaharia commented on HADOOP-3150:
---------------------------------------
Hi Devaraj,
Thanks for pinging me about this. Here is my thought on this problem: There's
only one right way to schedule cleanup tasks, namely as soon as the job
finishes, so schedulers should not be aware of this process. The role of a
scheduler is to decide which jobs get to run when there are several choices,
while initialization, cleanup, etc are responsibilities of the JobTracker. So
the solution I'd suggest is to assign cleanup tasks from the JobTracker code
itself before it calls the scheduler. That is, if(there are tasks to clean up)
{ send heartbeat response to launch cleanup task } else { call
taskScheduler.assignTasks }. This is clean and reduces the work needed to write
a scheduler and the potential bugs that can arise. It's much less hacky than
making obtainNewMap/Reduce return these tasks.
I'd also think about using a separate type of slot for cleanup tasks, i.e.
having them not count towards map and reduce slots. Presumably there will never
be a huge backlog of cleanup tasks, and these tasks won't impact performance of
running maps and reduces. This will let cleanup tasks run ASAP even when a node
is running long maps or reduces. However, if this is complicated to implement,
then don't worry about it.
> Move task file promotion into the task
> --------------------------------------
>
> Key: HADOOP-3150
> URL: https://issues.apache.org/jira/browse/HADOOP-3150
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Amareshwari Sriramadasu
> Fix For: 0.19.0
>
> Attachments: 3150.patch, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt,
> patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt, patch-3150.txt
>
>
> We need to move the task file promotion from the JobTracker to the Task and
> move it down into the output format.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
