Logged this request here to track progress... https://issues.apache.org/jira/browse/CRUNCH-579
On Fri, Nov 20, 2015 at 12:44 PM, Josh Wills <[email protected]> wrote: > Yeah, that does seem sort of crappy. Let me see what I can do. > > On Thu, Nov 19, 2015 at 10:39 PM, [email protected] < > [email protected]> wrote: > >> Hello, >> >> So we wrote a custom Crunch Target to write data to a particular location >> which included providing RecordWriters, OutputFormat, OutCommitters and >> such. Now we wanted to add Counters to get a count of how much data our >> reducers are writing. A most obvious design was to use the >> TaskAttemptContext that gets passed to the RecordWriter and use the >> getCounter() method on it to manipulate the counters. However, that did not >> work as expected and even though the counters were getting incremented, >> they did not show up on the Resource Manager UI. On further investigation I >> found that, cruch mangles this context object to add in a named output via >> the CrunchOutputs class[1] which basically makes the counters useless >> within the recordwriter class. >> >> Would it be a feasible enhancement for the CrunchOutputs to pass in the >> original base context object along with the modified one with named outputs >> so that it can be used for counters? Any other suggestions are most welcome >> as well. >> >> Thanks, >> Swarnim >> >> [1] >> https://github.com/cloudera/crunch/blob/cdh5.4.8-release/crunch-core/src/main/java/org/apache/crunch/io/CrunchOutputs.java#L210-L232 >> > >
