I see the returning of null for a PipelineResult as a bug in Crunch, and logged CRUNCH-443 [1] a while back (although it's still open). It's not totally clear what the best thing to do would be in such a situation, but I think that the returning of null should definitely change.
- Gabriel 1. https://issues.apache.org/jira/browse/CRUNCH-443 On Wed, Dec 31, 2014 at 6:55 AM, Jinal Shah <jinalshah2...@gmail.com> wrote: > Hi Kamal, > > PipelineResult can be null. This will happen when there is a job failure in > the pipeline(in almost all the cases I have seen). Not propagating to the > launcher is because internally Crunch handles any exceptions and return a > NULL object instead of throwing the exception. You could get the exception > if you look at jobs details page. You could possibly raise the argument to > throw an exception when a jobs from pipeline.run and pipeline.done. > > Thanks, > Jinal > > On Tue, Dec 30, 2014 at 8:38 PM, Kamal Ananthaneni < > kamal.ananthan...@gmail.com> wrote: > >> What are the scenarios in which pipeline.run() or pipeline.done() returns a >> PipelineResult of null? >> >> We have a pipeline that scans from HBase, does some processing runs the >> pipeline and does some additional processing afterwords. >> >> We noticed that if the Scan from HBase failed due to Socket Timeouts or >> other exceptions, the PipelineResult from pipeline.run() was null. >> >> We were not treating PipelineResult == null as an error scenario and were >> only checking for PipelineResult.status if PipelineResult is != null and >> hence the error was not getting propagated to the launcher job. >> >> In this case the error would have propagated and failed our launcher job >> had we treated the PipelineResult == null as an Error scenario. >> >> Is the assumption that PipelineResult == null on Pipeline.run() or >> Pipeline.done() indicates an error a valid one? In what other scenarios can >> a PipelineResult be null? >> >> Thanks, >> Kamal >>
