Thanks for the reply.
Got a couple of questions...
I saw that I could get the StatusReporter from the Mapper.Context.
But the API doesn't have a way to get the StatusReport from the Mapper.Context.
If you're going to go to the Mapper.Context you can get the counters.
And you have a getCounter(String, String). Then you can always increment it.
But this kind of begs a question which isn't documented...
If I have a mapper.Context and I say getCounter("Foo","Foobar Count");
What happens if there isn't a counter ("Foo","Foobar Count") ?
What I mean is how do you add a dynamic counter or if I ask for a counter that
doesn't already exist, will it create it?
(This isn't documented... and its not safe to assume anything.)
Sorry, but dynamic counters isn't really well documented.
-Mike
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Anthony Urso
Sent: Monday, July 19, 2010 7:11 PM
To: [email protected]
Subject: Re: Dynamic counters in Hadoop M/R jobs...
The new Mapper class passes a Map.Context object to the map() method.
From this you can get a StatusReporter object which can produce a
named Counter object and increment it.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Mapper.Context.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/StatusReporter.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Counter.html
Alternatively, the old APIs are supposed to be temporarily
de-deprecated in the next stable release, so you can use them as
described by that book.
On Mon, Jul 19, 2010 at 11:22 AM, Michael Segel
<[email protected]> wrote:
>
> Hi,
>
> In looking at "The Definitive Guide" pgs 211-218 looking at the
> documentation for counters have a question about how to use the
>
> dynamic (String, String) methods to create/access counters.
>
> Looking at the IdentityTableMap class, the following map() method exists():
> public void map(ImmutableBytesWritable key,
> RowResult value,
>
> org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWritable,RowResult>
> output,
> org.apache.hadoop.mapred.Reporter reporter)
> throws IOExceptionThis method is deprecated in 20.5
>
> So how can you get the Reporter so you can use the
> 'reporter.incrCounter(String group,String counter, long amount)'
> method. (See pg 214 code fragment)
>
> I must be missing something.
>
> Thx
>
> -Mike
>
>
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with
> Hotmail.
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
The information contained in this communication may be CONFIDENTIAL and is
intended only for the use of the recipient(s) named above. If you are not the
intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication, or any of its contents, is
strictly prohibited. If you have received this communication in error, please
notify the sender and delete/destroy the original message and any copy of it
from your computer or paper files.
