pan3793 commented on PR #52091: URL: https://github.com/apache/spark/pull/52091#issuecomment-3232864792
> Since this is not a general config, can we make it k8s specific https://spark.apache.org/docs/latest/running-on-kubernetes.html#configuration? @yaooqinn Actually, compared to other Resource Managers, YARN is special, it does additional cleanup work when the AM main method exits. I tested Standalone cluster mode, it's similar to K8s, non-daemon threads also block driver JVM exits. I don't have env to test Mesos, but have a quick look at the codebase, I think it's similar to K8s. <img width="851" height="365" alt="image" src="https://github.com/user-attachments/assets/1eea3146-9e6a-41ed-bb83-75fc8c18593f"; /> For cases where the driver runs outside of the Resource Manager, e.g. local or client mode, non-daemon threads also block driver JVM exits. In summary, I think this feature is a general solution to tackle the non-daemon threads issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
