[
https://issues.apache.org/jira/browse/SPARK-59813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59813:
-----------------------------------
Labels: pull-request-available (was: )
> Dump thread stacks when SparkContext initialization times out in
> ApplicationMaster
> ----------------------------------------------------------------------------------
>
> Key: SPARK-59813
> URL: https://issues.apache.org/jira/browse/SPARK-59813
> Project: Spark
> Issue Type: Improvement
> Components: YARN
> Affects Versions: 3.3.2
> Reporter: yanbin.zhang
> Priority: Major
> Labels: pull-request-available
>
> When SparkContext initialization times out in YARN cluster mode
> (spark.yarn.am.waitTime),
> the ApplicationMaster fails the application with exit code 13 and only
> advises to
> "check earlier log output for errors". However, in real production incidents
> the driver
> thread is often blocked silently (e.g., a slow/contended node, or a blocking
> call that
> produces no logging) and there is no earlier log output at all, leaving
> operators with
> no way to tell where the driver thread was stuck.
> Production example: a Kyuubi Spark engine failed with
> "java.util.concurrent.TimeoutException: Futures timed out after [100000
> milliseconds]"
> (exit code 13). The driver stdout showed ~80 seconds of complete silence
> right after
> SparkEnv initialization finished. Cross-checking HDFS NameNode and Hive
> Metastore logs
> proved the driver issued zero external I/O during the hang, but without a
> thread dump
> there was no way to pinpoint the blocking frame.
> Proposal: when the timeout fires, log (1) the stack trace of the user
> application
> thread (driver), and (2) a full thread dump, before failing the application.
> The dump
> is best-effort so it never masks the original timeout error. This follows the
> same
> pattern as the existing spark.task.reaper.threadDump on the executor side.
> PR: https://github.com/apache/spark/pull/58991
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
