[
https://issues.apache.org/jira/browse/MAPREDUCE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779332#action_12779332
]
dhruba borthakur commented on MAPREDUCE-1218:
---------------------------------------------
The proposal is to include the following metrics to be reported by each
TackTracker in every heartbeat:
B1. available physical memory on this machine (in bytes)
B2. cumulative used cpu time (for all cores) since the machine is up (in
millisecond)
B3. cpu speed on this machine (in Hz)
B4. # of cpu cores on the machine
I agree with Hong that the base methods that parse the /proc files to extract
the above metrics be in the common subproject. That will allow both mapreduce
and hdfs to use the same routines to extract useful information. On the other
hand, when to extract these metrics and how best to use them (whether NN or JT)
is best left to the individual subproject, isn't it? In fact, we had earlier
embarked on keeping the reporting metrics reporting framework outside the
JT/TT, but Vinod mentioned via MAPREDUCE-961 that it best if we can integrate
them into the existing JT-TT framework, see focussed comment here:
https://issues.apache.org/jira/browse/MAPREDUCE-961?focusedCommentId=12765422&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12765422
@Hong: I am hoping that you agree with my proposal of keeping the routines in
common subproject (so that HDFS can use it) but sending the extracted values in
the JT-TT heartbeats. It makes integration with JT-TT easier and elegant.
> Collecting cpu and memory usage for TaskTrackers
> ------------------------------------------------
>
> Key: MAPREDUCE-1218
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1218
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Environment: linux
> Reporter: Scott Chen
> Assignee: Scott Chen
>
> The information can be used for resource aware scheduling.
> Note that this is related to MAPREDUCE-220. There the per task resource
> information is collected.
> This one collects the per machine information.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
