singer-bin opened a new pull request, #58991: URL: https://github.com/apache/spark/pull/58991
## What changes were proposed in this pull request? When SparkContext initialization times out in YARN cluster mode, the ApplicationMaster fails the application with exit code 13 and only advises to "check earlier log output for errors". However, in real production incidents the driver thread is often blocked *silently* (slow/contended node, a blocking call with no logging, etc.) and there is **no** earlier log output at all, leaving operators with no way to tell where the driver thread was stuck. This PR logs thread stack traces when the timeout fires, before failing the application: 1. The stack trace of the user application thread (driver) — usually the most relevant one; 2. A full thread dump of all threads for extra context. The dump is best-effort: any failure while capturing it is swallowed so it never masks the original timeout error. ## Why are the changes needed? Production example: a long-running SQL engine (Kyuubi) failed with `java.util.concurrent.TimeoutException: Futures timed out after [100000 milliseconds]` and exit code 13. The driver stdout showed ~80 seconds of complete silence right after `SparkEnv` initialization finished. Cross-checking the HDFS NameNode and Hive Metastore logs proved the driver issued zero external I/O during the hang, so the usual suspects (slow NN/HMS calls) were all ruled out — but without a thread dump there was still no way to pinpoint the blocking frame. With this change, the AM log would directly show what the driver thread was blocked on. ## Does this PR introduce any user-facing change? No. It only adds error-level log output on an application-failure path. ## How was this patch tested? - Built the module and its dependency chain: `./build/mvn -Pyarn -pl resource-managers/yarn -am package -DskipTests` - `scalastyle:check` on `resource-managers/yarn` and `checkstyle:check` on `common/utils-java` both pass - `ApplicationMasterSuite` passes The `runDriver` timeout path has no existing test harness (it requires a live YARN RM), and this is a logging-only change on a failure path, so no new test is added. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
