The counts are different in the different StageResults. Running in MemPipeline gives only one counter result. I didn’t note it down at the time, but I think the MemPipeline count was the same as summing the MRPipeline results.
T From: Josh Wills <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Wednesday 25 March 2015 21:26 To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Re: counters in multistage pipeline My guess is yes, it's okay to just sum the results, but I'll double check the code- I'm assuming the counts are different in the different StageResults? Nothing that does an automatic merge, but agree that it would be useful. Post a patch? Maybe a new method on PipelineResult? On Wed, Mar 25, 2015 at 10:01 AM Tycho Lamerigts <[email protected]<mailto:[email protected]>> wrote: I have one DoFn that increments a counter (and a bunch of other DoFns that don’t use counters). When my DoFn with counter runs in a MRPipeline that returns multiple StageResults, I notice that my counter appears with a non-zero value in 2 StageResults. Is it correct to just sum up these values? Is there any utility class to this automatically for all counters in PipelineResult? T ________________________________ The information in this e-mail is intended only for the person or entity to which it is addressed. It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it. If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments. Messages and attachments are scanned for all known viruses. Always scan attachments before opening them. ________________________________ The information in this e-mail is intended only for the person or entity to which it is addressed. It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it. If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments. Messages and attachments are scanned for all known viruses. Always scan attachments before opening them.
