[
https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748405#action_12748405
]
Devaraj Das commented on MAPREDUCE-901:
---------------------------------------
bq. 1. I'd replace the array of longs with explicit fields.
Owen, is this to avoid the array indexing? I kept it this way since the
TaskCounter enum already existed, and it seemed simple to have a one-to-one
mapping from the enum to the array. Also, in the future if we add another
element to the TaskCounter enum, we don't have to explicitly modify TaskMetrics
since it would automatically handle that..
bq. 2. I'd drop the booleans for whether it has been used.
I kept this so as to retain the existing semantics where an unused counter is
not displayed to the user (for e.g., reduce-shuffle-bytes is not displayed as
part of map tasks' counters). But I am okay to do without the booleans..
bq. 3. The serialization should just be to write the fields in order using
vints.
If we remove the booleans, then it will be just writing the long array elements
which is already written as vlongs... If we decide to switch to explicit fields
for the longs then this part would change slightly..
bq. 4. updateFrameworkCounters should just update the counters by using the
enum values explicitly.
Again, as i pointed out in (1) above, it helps to iterate over TaskCounter enum.
bq. 5. updateFrameworkCounters should set, not increment the related counter.
Makes sense
bq. 6. TaskMetrics should be a field in the TaskStatus.
It already is. A little bit of cleanup may be required w.r.t passing the object
around in the TaskStatus methods..
> Move Framework Counters into a TaskMetric structure
> ---------------------------------------------------
>
> Key: MAPREDUCE-901
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-901
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.21.0
> Reporter: Owen O'Malley
> Assignee: Devaraj Das
> Fix For: 0.21.0
>
> Attachments: 901_1.patch, 901_1.patch
>
>
> I think we should move all of the Counters that the framework updates into a
> single class called TaskMetrics. TaskMetrics would have specific fields for
> each of the metrics like input records, input bytes, output records, etc.
> It would both reduce the serialized size of the heartbeats (by shrinking the
> Counters down to just the user's counters) and decrease the latency for
> updates to the JobTracker (since Counters are sent at most 1/minute instead
> of 1/heartbeat).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
