Can you identify the error case and call System.exit ? It'll get retried on another executor, but as long as that one fails the same way...
If you can identify the error case at the time you're doing database interaction and just prevent data being written then, that's what I typically do. On Tue, Jan 24, 2017 at 7:50 AM, Erwan ALLAIN <eallain.po...@gmail.com> wrote: > Hello guys, > > I have a question regarding how spark handle failure. > > I’m using kafka direct stream > Spark 2.0.2 > Kafka 0.10.0.1 > > Here is a snippet of code > > val stream = createDirectStream(….) > > stream > .map(…) > .forEachRDD( doSomething) > > stream > .map(…) > .forEachRDD( doSomethingElse) > > The execution is in FIFO, so the first action ends after the second starts > so far so good. > However, I would like that when an error (fatal or not) occurs during the > execution of the first action, the streaming context is stopped immediately. > It's like the driver is not notified of the exception and launch the second > action. > > In our case, the second action is performing checkpointing in an external > database and we do not want to checkpoint if an error occurs before. > We do not want to rely on spark checkpoint as it causes issue when upgrading > application. > > Let me know if it’s not clear ! > > Thanks ! > > Erwan --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
