So far I've only tried this in the SparkPipeline. In MemPipeline the entire JVM dies, so we don't get to determine success or failure.
On Mon, Jan 19, 2015 at 10:47 AM, Josh Wills <[email protected]> wrote: > No, that's not good, we should fix that. Is it only in the SparkPipeline > that the situation occurs? > > On Mon, Jan 19, 2015 at 8:28 AM, Peter Dolan <[email protected]> wrote: > >> Hi Crunchers, >> >> At Nuna we've been using Crunch extensively, and I'm really thrilled with >> it. It's excellent. There are of course some rough edges though. >> >> Today I ran into some exceptions being thrown in the Spark pipeline, and >> am curious why they weren't resulting in the PipelineResult reporting >> failure. In particular, my spark pipeline (running with a local spark >> instance, that is with the spark master set to "local[16]") failed with an >> IOException when the machine ran out of space in /tmp/. The PipelineResult >> retrieved by Pipeline#done returned true from PipelineResult#succeeded. >> >> I've seen this in a couple other contexts, for example when a MapFn threw >> an exception within MapFn#map, which did not result in a false success >> value. >> >> Is this expected / intended behavior? Should I be getting at the success >> or failure of the execution some other way? >> >> Thanks! >> - Peter >> > >
