Thanks Josh! On Tue, Jan 20, 2015 at 9:58 AM, Josh Wills <[email protected]> wrote:
> Okay, created https://issues.apache.org/jira/browse/CRUNCH-488 to track > it. Should get a patch together by tmrw. > > J > > On Mon, Jan 19, 2015 at 4:57 PM, Peter Dolan <[email protected]> wrote: > >> So far I've only tried this in the SparkPipeline. In MemPipeline the >> entire JVM dies, so we don't get to determine success or failure. >> >> On Mon, Jan 19, 2015 at 10:47 AM, Josh Wills <[email protected]> >> wrote: >> >>> No, that's not good, we should fix that. Is it only in the SparkPipeline >>> that the situation occurs? >>> >>> On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <[email protected]> >>> wrote: >>> >>>> Hi Crunchers, >>>> >>>> At Nuna we've been using Crunch extensively, and I'm really thrilled >>>> with it. It's excellent. There are of course some rough edges though. >>>> >>>> Today I ran into some exceptions being thrown in the Spark pipeline, >>>> and am curious why they weren't resulting in the PipelineResult reporting >>>> failure. In particular, my spark pipeline (running with a local spark >>>> instance, that is with the spark master set to "local[16]") failed with an >>>> IOException when the machine ran out of space in /tmp/. The PipelineResult >>>> retrieved by Pipeline#done returned true from PipelineResult#succeeded. >>>> >>>> I've seen this in a couple other contexts, for example when a MapFn >>>> threw an exception within MapFn#map, which did not result in a false >>>> success value. >>>> >>>> Is this expected / intended behavior? Should I be getting at the >>>> success or failure of the execution some other way? >>>> >>>> Thanks! >>>> - Peter >>>> >>> >>> >> > > > -- > Director of Data Science > Cloudera <http://www.cloudera.com> > Twitter: @josh_wills <http://twitter.com/josh_wills> >
