[
https://issues.apache.org/jira/browse/HADOOP-2248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley updated HADOOP-2248:
----------------------------------
Attachment: counter-speedup.patch
This patch speed map/reduce up by about 10% overall wallclock. The changes are:
1. Introduce a cache from the counter enum directly to the counter using
identity hash map.
2. Change the api of counters so that you can iterate through them using for
loops.
3. Change the api of counters so that applications can keep a reference to the
counter.
4. Change the serialization to use Text.writeString instead of UTF8.writeString.
5. Change the map/reduce framework to keep reference to its internal counters
(counts of map, combiner and reduce inputs and outputs)
6. Adds to the counters api to do string based incrementing to support
HADOOP-1915.
> Word count example is spending 24% of the time in incrCounter
> -------------------------------------------------------------
>
> Key: HADOOP-2248
> URL: https://issues.apache.org/jira/browse/HADOOP-2248
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Reporter: Owen O'Malley
> Assignee: Owen O'Malley
> Fix For: 0.16.0
>
> Attachments: counter-speedup.patch
>
>
> When running under the local runner with the local file system, incrementing
> the user counters is consuming 24% of the cpu.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
