Re: spark on k8s driver pod exception

Sure, that is expected, see the "How it works" section in "Running Spark on Kubernetes" page , quote: When the application completes, the executor pods terminate and are cleaned > up, but the driver pod persists logs and

Re: spark on k8s driver pod exception

when driver pod throws exception , driver pod still running ？ kubectl logs wordcount-e3141c7834d3dd68-driver 21/03/15 07:40:19 DEBUG Analyzer$ResolveReferences: Resolving 'value1 to 'value1 Exception in thread "main" org.apache.spark.sql.AnalysisException: cannot resolve '`value1`' given

Re: spark on k8s driver pod exception

> but the spark-submit log still running Set the "spark.kubernetes.submission.waitAppCompletion" config to false to change that. As the doc says: "spark.kubernetes.submission.waitAppCompletion" : In cluster mode, whether to wait for the application to finish before exiting the launcher process.

Re: spark on k8s driver pod exception

For getting the logs please read Accessing Logs part of the *Running Spark on Kubernetes* page. For stopping and generic management of the spark application please read the Spark Application Management

spark on k8s driver pod exception

when run the code in k8s , driver pod throw AnalysisException , but the spark-submit log still running , then how to get the exception and stop pods ? val spark = SparkSession.builder().getOrCreate() import spark.implicits._ val df = (0 until 10).toDF("id").selectExpr("id %