Shutdown Spark application with failed state

2021-07-26 Thread Anil Dasari
Hello all, I am using Spark 2.x streaming with kafka. I noticed that spark streaming is processing subsequent micro-batches in case of failure as it takes a while to notify the driver about the error and interrupt streaming-executor thread. This is creating a problem as we are checkpointing

Shutdown Spark application with failed state

2021-07-26 Thread Anil Dasari
Hi Team, I am using Spark 2.x streaming with kafka. I noticed that spark streaming is processing subsequent micro-batches in case of failure as it takes a while to notify the driver about the error and interrupt streaming-executor thread. This is creating problem as we are checkpointing the