Shutdown Spark application with failed state

Hi Team, I am using Spark 2.x streaming with kafka. I noticed that spark streaming is processing subsequent micro-batches in case of failure as it takes a while to notify the driver about the error and interrupt streaming-executor thread. This is creating problem as we are checkpointing the

Shutdown Spark application with failed state

Hello all, I am using Spark 2.x streaming with kafka. I noticed that spark streaming is processing subsequent micro-batches in case of failure as it takes a while to notify the driver about the error and interrupt streaming-executor thread. This is creating a problem as we are checkpointing

Why PySpark with spark-submit throws error trying to untar --archives pyspark_venv.tar.gz

Hi, Maybe someone can shed some light on this. Running Pyspark job in minikube. Because it is PySpark the following two conf parameters are used: spark-submit --verbose \ --master k8s://$K8S_SERVER \ --deploy-mode cluster \ --name pytest \