singer-bin commented on PR #58991: URL: https://github.com/apache/spark/pull/58991#issuecomment-5809057057
Hi @dongjoon-hyun @pan3793 @yaooqinn, gentle ping for a review when you have a chance. This is a small diagnostics-only change: when `spark.yarn.am.waitTime` fires (exit code 13), the AM now logs the driver thread's stack plus a full thread dump before failing the application. Motivation: we hit this in production with a Kyuubi Spark engine — the driver went completely silent for ~80s after SparkEnv finished, and HDFS/Hive Metastore logs proved zero external I/O, but there was no way to see where the driver thread was blocked. This follows the same pattern as the existing `spark.task.reaper.threadDump` on the executor side. (The earlier CI failure in `pyspark-install` was an Apache CDN 503 while downloading a Spark release tarball — unrelated; the failed job has been re-triggered.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
