[
https://issues.apache.org/jira/browse/MAPREDUCE-961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12772802#action_12772802
]
Matei Zaharia commented on MAPREDUCE-961:
-----------------------------------------
bq. I see. So we may need to count the previous continuously launched map tasks
in canLauch task and return a false if there are too many map tasks launched in
a row. Is this correct?
Yes, either that or to change the scheduler to alternate between looking for a
map and looking for a reduce. Right now the logic in there is organized as:
{code}
for taskType in {MAP, REDUCE}:
while true:
if canLaunchTask(..., taskType):
try to find a job with pending task
if found a job:
launch task
else:
break
{code}
It should become something like this:
{code}
while true:
pick taskType to try next (if node has fewer maps than reduces, choose map;
else choose reduce)
if canLaunchTask(..., taskType):
try to find a job with pending task
if found a job:
launch task
else:
break
{code}
> ResourceAwareLoadManager to dynamically decide new tasks based on current
> CPU/memory load on TaskTracker(s)
> -----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-961
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: HIVE-961.patch, MAPREDUCE-961-v2.patch
>
>
> Design and develop a ResouceAwareLoadManager for the FairShare scheduler that
> dynamically decides how many maps/reduces to run on a particular machine
> based on the CPU/Memory/diskIO/network usage in that machine. The amount of
> resources currently used on each task tracker is being fed into the
> ResourceAwareLoadManager in real-time via an entity that is external to
> Hadoop.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
