Thanks Micah. We are going to use only MRPipeline and passing the TaskInputOutputContext enabled me to implement counters in the helper class.
On Thu, Feb 12, 2015 at 4:27 PM, Micah Whitacre <[email protected]> wrote: > Venkatesh, > There isn't really a great way without you exposing your implementation > of DoFn to the runtime container you are running in. The DoFn.increment > method is hiding if the "increment" call is being made to alter a MapReduce > counter or Spark. If you know that you will only ever run in a MapReduce > world then you could pull the TaskInputOutputContext and increment a > counter using that but that seems like an abstraction leak of a concept > only known to your selected Pipeline implementation. > > Micah > > On Thu, Feb 12, 2015 at 6:12 PM, venkatesh kavuluri < > [email protected]> wrote: > >> Hi All, >> >> I am trying to figure out the best way to call increment method (for >> counters) from a helper class invoked from DoFn#process. >> >> I have something like this. >> >> class A extends DoFn<> { >> >> Foo foo = nee Foo(); // helper class >> >> @Override >> public void process (S input, Emitter<T> emitter) { >> foo.processAndEmitEvents(bar, emitter); >> } >> >> I want to call DoFn#increment method from Foo#processAndEmitEvents. Is >> there a better way to go about it other than “Foo" extending “A” just to >> get hold of DoFn#increment. >> >> Thanks for the help. >> > >
