[
https://issues.apache.org/jira/browse/MAPREDUCE-901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Lu updated MAPREDUCE-901:
------------------------------
Attachment: FrameworkCounterGroup.java
Looking through the patches, it seems that the existing approach requires
sweeping changes to internal apis across pretty much all mapreduce components.
I'm proposing a new approach to this issue: refactor the counter framework
(while keeping the existing api) so that it's possible to have different
implementation of counter groups, one of which is the FrameworkCounterGroup
(see attached file.)
This way, we can achieve the benefit of more efficient implementation for
framework counters without changing the client code.
The problem is a little complicated due to need to support of both old and new
counters interface (both marked public and stable.) I'm working on a more
complete patch to minimize the code duplication between new and old counter
code. But I'm confident that the resulting code will be more concise and
general (to support any future framework counters in different groups.) than
the existing approach.
> Move Framework Counters into a TaskMetric structure
> ---------------------------------------------------
>
> Key: MAPREDUCE-901
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-901
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: task
> Affects Versions: 0.21.0
> Reporter: Owen O'Malley
> Assignee: Luke Lu
> Attachments: 901_1.patch, 901_1.patch, FrameworkCounterGroup.java,
> MAPREDUCE-901.patch, MAPREDUCE-901.patch
>
>
> I think we should move all of the Counters that the framework updates into a
> single class called TaskMetrics. TaskMetrics would have specific fields for
> each of the metrics like input records, input bytes, output records, etc.
> It would both reduce the serialized size of the heartbeats (by shrinking the
> Counters down to just the user's counters) and decrease the latency for
> updates to the JobTracker (since Counters are sent at most 1/minute instead
> of 1/heartbeat).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
