Thanks Jinal and Gabriel for the prompt response! Since crunch-443 is not yet resolved, we will make the change on our side to handle pipelineResult == null as a failure.
Thanks, Kamal On Wed, Dec 31, 2014 at 3:48 AM, Gabriel Reid <gabriel.r...@gmail.com> wrote: > I see the returning of null for a PipelineResult as a bug in Crunch, > and logged CRUNCH-443 [1] a while back (although it's still open). > It's not totally clear what the best thing to do would be in such a > situation, but I think that the returning of null should definitely > change. > > - Gabriel > > > 1. https://issues.apache.org/jira/browse/CRUNCH-443 > > On Wed, Dec 31, 2014 at 6:55 AM, Jinal Shah <jinalshah2...@gmail.com> > wrote: > > Hi Kamal, > > > > PipelineResult can be null. This will happen when there is a job failure > in > > the pipeline(in almost all the cases I have seen). Not propagating to the > > launcher is because internally Crunch handles any exceptions and return a > > NULL object instead of throwing the exception. You could get the > exception > > if you look at jobs details page. You could possibly raise the argument > to > > throw an exception when a jobs from pipeline.run and pipeline.done. > > > > Thanks, > > Jinal > > > > On Tue, Dec 30, 2014 at 8:38 PM, Kamal Ananthaneni < > > kamal.ananthan...@gmail.com> wrote: > > > >> What are the scenarios in which pipeline.run() or pipeline.done() > returns a > >> PipelineResult of null? > >> > >> We have a pipeline that scans from HBase, does some processing runs the > >> pipeline and does some additional processing afterwords. > >> > >> We noticed that if the Scan from HBase failed due to Socket Timeouts or > >> other exceptions, the PipelineResult from pipeline.run() was null. > >> > >> We were not treating PipelineResult == null as an error scenario and > were > >> only checking for PipelineResult.status if PipelineResult is != null and > >> hence the error was not getting propagated to the launcher job. > >> > >> In this case the error would have propagated and failed our launcher job > >> had we treated the PipelineResult == null as an Error scenario. > >> > >> Is the assumption that PipelineResult == null on Pipeline.run() or > >> Pipeline.done() indicates an error a valid one? In what other scenarios > can > >> a PipelineResult be null? > >> > >> Thanks, > >> Kamal > >> >
