[
https://issues.apache.org/jira/browse/MAPREDUCE-771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12733000#action_12733000
]
Hemanth Yamijala commented on MAPREDUCE-771:
--------------------------------------------
Looking at usage of countMapTasks (and similar for reduces), all schedulers
except capacity scheduler count tasks instead of slots. This is fine for now
since no one else uses memory based scheduling to the best of my knowledge.
Ideally they should count slots so that whenever they support memory based
scheduling they get the right behavior.
The other usage is that these values are set in ClusterStatus. As far as I can
see, this is not a problem and looks fine to me because the usage is looking at
global map tasks, rather than number of slots on a machine. I have not checked
the usage in test cases.
> Setup and cleanup tasks remain in UNASSIGNED state for a long time on
> tasktrackers with long running high RAM tasks
> -------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-771
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-771
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Hemanth Yamijala
> Fix For: 0.21.0
>
>
> When a high RAM job's task is scheduled on a tasktracker, the number of slots
> it occupies will be more than one. If enough such tasks are scheduled, there
> can come a situation when there are no more slots free on the node but the
> number of tasks running is less than the number of slots. The jobtracker
> currently schedules a setup or cleanup task based on how many tasks are
> running on the system, rather than slots. As a result of this, it can
> schedule a setup or cleanup task on a tasktracker without any free slots. If
> the high RAM job's tasks are long running, this will significantly delay the
> running of the setup or cleanup task, and thus the entire job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
