[
https://issues.apache.org/jira/browse/MAPREDUCE-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090015#comment-13090015
]
nandan commented on MAPREDUCE-2653:
-----------------------------------
In response to Allen Wittenauer's question:
How does this method work when the tasks are IO intensive?
Monitoring system on every TT categorizes and stores each task it runs, into
CPU-Intensive and CPU-NonIntensive lists (this includes Memory as well IO
intensive tasks) and generates job request by selecting jobs from these lists
one by one alternately, considering current cpu-idle time and cpu utilization
of the task. Request consists of list of jobs whose map tasks TT can run as
extra tasks. This request is submitted to JT through heartbeat, which processes
jobs from the request one by one.
So currently I am treating IO and Memory processes as same.
> dynamic map slots (in addition to predifined) on each node which allows to
> execute cpu intensive jobs along with memory intensive jobs thereby reducing
> wastage of cpu cycles
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2653
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2653
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker, tasktracker
> Affects Versions: 0.20.203.0
> Environment: linux
> Reporter: nandan
> Labels: map, scheduler, tasks
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> I have introduced process monitoring system inside tasktracker, which
> analyses the cpu and memory utilization of each map task and allows me to
> increase/decrease maximum number of map slots dynamically on each node. With
> this I can combine cpu intensive jobs along with memory intensive jobs,
> thereby reducing the cpu idle time.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
