[
https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779829#action_12779829
]
Hong Tang commented on MAPREDUCE-1218:
--------------------------------------
I realized you are looking for some quick solutions. So I am fine with your
suggestions.
>From a long term perspective, I think we ought to think more on the overall
>architecture of hadoop:
In some cluster management systems I know of, typically load management and
service discovery are structured as a standalone layer, and computation
framework and storage systems like MR, HDFS would be services built on top of
that. There are many advantages of such a design over the current architecture,
two of which on top of head are:
- the load information may be shared across multiple tenants that share the
same resources, and they can coordinate on load balance objectives.
- code is more modular and easier to maintain
There are also a lot of researches on dispersing load information across nodes
and using the load information wisely, particularly in situations where load
information fluctuate quite frequently (so once every 20s heartbeat could be
too coarse-grained). There are a series of papers in these areas from the
research group where I did my Ph.D.: http://www.cs.ucsb.edu/projects/neptune/
> Collecting cpu and memory usage for TaskTrackers
> ------------------------------------------------
>
> Key: MAPREDUCE-1218
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Environment: linux
> Reporter: Scott Chen
> Assignee: Scott Chen
>
> The information can be used for resource aware scheduling.
> Note that this is related to MAPREDUCE-220. There the per task resource
> information is collected.
> This one collects the per machine information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
