pan3793 commented on PR #46889: URL: https://github.com/apache/spark/pull/46889#issuecomment-2153244615
> This raises an interesting question of whether my new flag should also be calling `sc.stop()` if it was not stopped by a user. > although "stop the context then call System.exit()" is probably a reasonable behavior for most cases, there might be a long-tail of less common use-cases where users explicitly don't want any automatic cleanup or shutdown. I believe `System.exit` and `SparkContext.stop` are different things, and we'd better have two flags to cover all possible cases, hopefully. Of course, if both flags are enabled, call `SparkContext.stop` first then `System.exit`. > ... shutdown hooks (some of which might be implicitly (and possibly incorrectly) assuming that the `SparkContext` will have already been stopped before their hooks run). ... and versus, I do see that some users write shutdown hooks rely on the priority to assume that the execution happens before/after SparkContext shutdown. > I'm slightly hesitant to want to change those defaults for fear of breaking someone's use-case. Yeah, I understand. Well, on the other hand, Spark 4.0 may be a good opportunity(at least better than minor versions like 4.1, 4.2 ...) to make such a breaking change with a more reasonable and consistent(as mentioned above, the default K8s's shutdown behavior is different from other platforms), as long as it has flags to allow users change back to previous behaviors. > non-daemon threads might cause logically-completed spark-submit jobs to hang rather than completing. And IIRC, in Spark on YARN cluster mode(the most popular cases in my company and customers), the AM will call `System.exit` thus there are no such issues, but it does happen on K8s mode. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
