[ http://issues.apache.org/jira/browse/HADOOP-492?page=comments#action_12442211 ] arkady borkovsky commented on HADOOP-492: -----------------------------------------
[[ Old comment, sent by email on Wed, 30 Aug 2006 16:40:37 -0700 ]] One of the intentions of Global Counters is for use in application code. E.g. if I count words in a the input, I'd like to know the total number of words, not just the count for each word. With vanilla MapReduce, I need a separate job to do the totals. Global Counter would let me to do this during the first job. > Global counters > --------------- > > Key: HADOOP-492 > URL: http://issues.apache.org/jira/browse/HADOOP-492 > Project: Hadoop > Issue Type: New Feature > Components: mapred > Reporter: arkady borkovsky > > It would be nice to have map / reduce job keep aggregated counts for > arbitrary events occuring in its tasks -- the numer of records processed, the > numer of exceptions of a specific type, the number of sentences in passive > voice, whatever the jobs finds useful. > This can be implemented by tasks periodically sending <name, value> pairs to > the jobtracker (in some implementations such messages are piggy-backed on the > heartbeats), so that the job tracker stores all the latests values from each > task and aggregates them on a request. It should also make the aggregated > values available at the job end. The value for a task would be flushed when > the task fails. > #491 and #490 may be related to this one. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
