[
https://issues.apache.org/jira/browse/MAPREDUCE-2037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2037:
-------------------------------------
Release Note:
Capture intermediate task resource consumption information:
* Time taken so far
* CPU load [either at the time the data are taken, or exponentially smoothed]
* Memory load [also either at the time the data are taken, or exponentially
smoothed]
This would be taken at intervals that depend on the task progress plateaus. For
example, reducers have three progress ranges - [0-1/3], (1/3-2/3], and
(2/3-3/3] - where fundamentally different activities happen. Mappers have
different boundaries that are not symmetrically placed [0-9/10], (9/10-1]. Data
capture boundaries should coincide with activity boundaries. For the state
information capture [CPU and memory] we should average over the covered
interval.
Editorial pass over hadoop-0.23 content.
> Capturing interim progress times, CPU usage, and memory usage, when tasks
> reach certain progress thresholds
> -----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2037
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2037
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Dick King
> Assignee: Dick King
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2037.patch, MAPREDUCE-2037.patch
>
>
> We would like to capture the following information at certain progress
> thresholds as a task runs:
> * Time taken so far
> * CPU load [either at the time the data are taken, or exponentially
> smoothed]
> * Memory load [also either at the time the data are taken, or
> exponentially smoothed]
> This would be taken at intervals that depend on the task progress plateaus.
> For example, reducers have three progress ranges -- [0-1/3], (1/3-2/3], and
> (2/3-3/3] -- where fundamentally different activities happen. Mappers have
> different boundaries, I understand, that are not symmetrically placed. Data
> capture boundaries should coincide with activity boundaries. For the state
> information capture [CPU and memory] we should average over the covered
> interval.
> This data would flow in with the heartbeats. It would be placed in the job
> history as part of the task attempt completion event, so it could be
> processed by rumen or some similar tool and could drive a benchmark engine.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
