[
https://issues.apache.org/jira/browse/HADOOP-4474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641709#action_12641709
]
Vivek Ratan commented on HADOOP-4474:
-------------------------------------
bq. But note that the iteration is done only when a TT asks for a task and not
per a heartbeat.
Fair enough. But using a linked list like structure to track these special
tasks seems simple enough. It also gives you a big benefit that you can
maintain, and hence schedule, setup/cleanup tasks in a particular order. You'd
ideally like to run setup tasks of jobs in the order in which they were
submitted. An iteration over a hashmap is random. Using a separate sorted
structure for setup/cleanup tasks can give you the ordering you want, which I
think is quite valuable. There shouldn't be too many of these tasks (one each
per job), so your memory footprint is relatively low.
> JT should not iterate through all jobs in every heartbeat to find a cleanup
> or setup task
> -----------------------------------------------------------------------------------------
>
> Key: HADOOP-4474
> URL: https://issues.apache.org/jira/browse/HADOOP-4474
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Vivek Ratan
>
> On every heartbeat, the JT first looks to see if it can run a setup or
> cleanup task, before calling a Scheduler to get a Map or Reduce task. The JT
> maintains a hashmap of JobInProgress objects (which can be waiting, running,
> or completed). It iterates through this hashmap on each heartbeat to find a
> setup or cleanup task. This linear search can be be very expensive,
> especially with large clusters where the number of jobs is high. There are
> lots of obvious ways to cut down on this linear search.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
