[
https://issues.apache.org/jira/browse/CRUNCH-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13783430#comment-13783430
]
Micah Whitacre commented on CRUNCH-272:
---------------------------------------
To which persona would you want this feature to be available? As a programmer
creating the Pipeline + Java Action that Oozie is launching? As an Oozie
workflow owner assembling multiple Java Actions and therefore not able to
access Crunch APIs?
The reason I ask is that some of these capabilities are already available in
certain situations. Also since Crunch has no concept of Oozie and Oozie no
concept of Crunch it is very difficult to try and tie the two together.
> Unable to correlate crunch jobs within Oozie
> --------------------------------------------
>
> Key: CRUNCH-272
> URL: https://issues.apache.org/jira/browse/CRUNCH-272
> Project: Crunch
> Issue Type: Improvement
> Reporter: Mike Zimmerman
>
> I'm not really sure if this should be logged to Oozie or to Crunch, so please
> feel free to move as needed.
> I would like to request a way to decorate map/reduce jobs that are spawned by
> a Crunch pipeline so that I can programmatically determine their origin. The
> primary use case for this is integration with Oozie. Oozie launches a single
> map job to run a java action (in our case this java action runs a crunch
> job). Traceability from this original "launcher" job to the jobs created by
> the crunch job is impossible without trolling logs. This leaves a big black
> hole for the system operator to assess the performance/impact of these jobs.
> My initial thought was to provide a simple way to indicate a correlationId or
> similar on a map/reduce job and then make it accessible within Oozie to query
> for. Obviously, that request would have to come after the correlation
> feature was available within map/reduce.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
