igorcalabria opened a new pull request #25785: [SPARK-27812][CORE][Kubernetes] Explicit System.exit after job's main URL: https://github.com/apache/spark/pull/25785 ### What changes were proposed in this pull request? Explicit calling `System.exit` after user's main code runs. ### Why are the changes needed? https://issues.apache.org/jira/browse/SPARK-27812 https://issues.apache.org/jira/browse/SPARK-27927 If there are non-daemon threads running, the JVM won't call `ShutdownHook` after the driver's main exits. This means that any job running on kubernetes that doesn't explicitly call `SparkSession#stop` will hang. I believe that expecting users to include this to every job is unreasonable since they also need to remember to add an `UncaughtExceptionHandler`. If there's no exception handler, any exception thrown on the driver's side will also hang the process. Since I'm not that familiar with spark's codebase, this could be a terrible idea and I'm hopping that some of you guys could propose a better solution if that's the case. My educated guess is that there's no expectation that the application will continue to run after the declared main, the only difference is that we're now calling `System.exit` so shutdown hooks run independently of random non daemon threads. ### Does this PR introduce any user-facing change? I'm guessing that no. It does not introduce something that the user should notice(besides the fix) ### How was this patch tested? Took `SparkPI` example and removed the `spark.stop()` call. Expected behaviour is that the driver exits after the job, but it doesn't. This patch fixed this.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
