[
https://issues.apache.org/jira/browse/HADOOP-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12701377#action_12701377
]
Khaled Elmeleegy commented on HADOOP-5632:
------------------------------------------
I think that the work division between the light weight and the heavy weight
heartbeats needs more thought. If running tasks status updates can be done
cheaply, we can piggyback these updates on the lightweight heartbeats. On
one extreme, each light weight heartbeat will have status updates of all the
running tasks. On the other extreme, tasks status updates are only passed
with the heavy weight heartbeats. Or somewhere in the middle, where some
light weight heartbeats can have status updates of some of the running
tasks.
I was thinking of having the heavy weight heartbeats have a low frequency,
like every few minutes. If this would create a problem for the jobtracker as
it will not be as up-to-date with the running tasks' status as we'd need,
then running tasks status updates will need to be somehow piggybacked on the
light weight heartbeats.
I think that once we agree on what's done in each heartbeat type, we can
then decide on what to call each type of heartbeats.
Khaled
> Jobtracker leaves tasktrackers underutilized
> --------------------------------------------
>
> Key: HADOOP-5632
> URL: https://issues.apache.org/jira/browse/HADOOP-5632
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.18.0, 0.18.1, 0.18.2, 0.18.3, 0.19.0, 0.19.1, 0.20.0
> Environment: 2x HT 2.8GHz Intel Xeon, 3GB RAM, 4x 250GB HD linux
> boxes, 100 node cluster
> Reporter: Khaled Elmeleegy
> Attachments: hadoop-khaled-tasktracker.10s.uncompress.timeline.pdf,
> hadoop-khaled-tasktracker.150ms.uncompress.timeline.pdf, jobtracker.patch,
> jobtracker20.patch
>
>
> For some workloads, the jobtracker doesn't keep all the slots utilized even
> under heavy load.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
