[
https://issues.apache.org/jira/browse/CRUNCH-579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Micah Whitacre updated CRUNCH-579:
----------------------------------
Attachment: CRUNCH-579.patch
So this just wraps some methods from the original context vs passing both into
the writers. Thoughts?
> Support Counters from Custom RecordWriters
> ------------------------------------------
>
> Key: CRUNCH-579
> URL: https://issues.apache.org/jira/browse/CRUNCH-579
> Project: Crunch
> Issue Type: Improvement
> Components: Core
> Reporter: Micah Whitacre
> Assignee: Josh Wills
> Attachments: CRUNCH-579.patch
>
>
> A consumer mentioned this on the mailing list:
> {quote}
> So we wrote a custom Crunch Target to write data to a particular location
> which included providing RecordWriters, OutputFormat, OutCommitters and such.
> Now we wanted to add Counters to get a count of how much data our reducers
> are writing. A most obvious design was to use the TaskAttemptContext that
> gets passed to the RecordWriter and use the getCounter() method on it to
> manipulate the counters. However, that did not work as expected and even
> though the counters were getting incremented, they did not show up on the
> Resource Manager UI. On further investigation I found that, cruch mangles
> this context object to add in a named output via the CrunchOutputs class[1]
> which basically makes the counters useless within the recordwriter class.
> Would it be a feasible enhancement for the CrunchOutputs to pass in the
> original base context object along with the modified one with named outputs
> so that it can be used for counters? Any other suggestions are most welcome
> as well.
> {quote}
> http://mail-archives.apache.org/mod_mbox/crunch-user/201511.mbox/%3CCAHnpetQpcSqFhWFZ9ZJg6DkN02jeC%3DLpvJ0%2BVSP%2BoA%2B8c0DK%2Bw%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
