Thanks Josh. Let me know what you think. I have a patch for this (so pipelines don't succeed if one of the stages fails).
-- Joe On Thu, Jun 6, 2013 at 1:49 PM, Josh Wills <[email protected]> wrote: > Definitely sounds like a bug. Will take a look when I get on this plane. > On Jun 6, 2013 12:36 PM, "Joseph Adler" <[email protected]> wrote: > >> Hi guys, >> >> I'm confused by how to tell if a Crunch Pipeline has succeeded. I suspect >> that the current implementation is not correct, but want to verify that >> before I file a Jira and submit a patch. >> >> Here's what I'm trying to do: I'd like to wrap a crunch job inside >> another Java program (so that I can execute it in a larger >> scheduling/workflow tool like Azkaban or Oozie). So, I'd like to build and >> execute a Crunch pipeline, then test if it succeeded. >> >> I'd like to do something like: >> >> PipelineResult result = pipeline.run(); >> if (!result.succeeded()) >> { >> throw new Exception("Job failed: " + result.toString() + "\n"); >> } >> return 0; >> >> >> Unfortunately, this doesn't work. Looking at the implementation of >> PipelineResult, here is how the succeeded method is implemented: >> >> public boolean succeeded() { >> return !stageResults.isEmpty(); >> } >> >> Looking at where the PipelineResult object is created (in MRExecutor), it >> looks like the list of stage results will be non-empty regardless of >> whether the job succeeded or not: >> >> ... >> result = new PipelineResult(stages); >> if (killSignal.getCount() != 0) { >> status.set(Status.KILLED); >> } else { >> status.set(result.succeeded() ? Status.SUCCEEDED : >> Status.FAILED); >> } >> ... >> >> So, even if one of the stages fails, the result of executing the pipeline >> will be "succeeded." >> >> I can't find an alternative way to test if a failure occurred during a >> pipeline execution. Is there something that I'm missing, or is the >> implementation incorrect? >> >> -- Joe >> >
