imback82 commented on pull request #28435: URL: https://github.com/apache/spark/pull/28435#issuecomment-624255510
> that certain sounds like a bug or misconfiguration somewhere else. You should not consistently be seeing a timeout in shutdown. Does it ever work then? Yes. In our production system, I rarely see this issue, but I found a configuration that could repro this behavior consistently. > I would like to know root cause before just making a change in spark for this. Yes, I agree. I am actively investigating this and will update this thread when I find out more. This PR was meant to see if it even makes sense to unregister AM outside the shutdown hook since shutdown hook relies on Hadoop library. > You would need to figure out what the function "anon$2' is in "ShutdownHook '$anon$2' timeout. `anon$2` is a hook that captures all the hooks registered in Spark: https://github.com/apache/spark/blob/86c4e4352565cf271a96b650f533ed89a725322c/core/src/main/scala/org/apache/spark/util/ShutdownHookManager.scala#L177-L181 In my repro, the first hook run from `anon$2` is the AM shutdown hook. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
