Fei Wang created SPARK-57204:
--------------------------------
Summary: capture Spark driver stderr to Kubernetes termination log
on failure
Key: SPARK-57204
URL: https://issues.apache.org/jira/browse/SPARK-57204
Project: Spark
Issue Type: Improvement
Components: Kubernetes
Affects Versions: 4.2.0
Reporter: Fei Wang
* On non-zero exit, write the last 4KB of the Spark driver's stderr to
{{/dev/termination-log}} so that {{kubectl describe pod}} surfaces the error in
the pod's *Last State > Message* field
* Driver stderr is streamed via a named pipe + awk ring buffer — fully visible
in {{kubectl logs}} with no unbounded {{/tmp}} disk growth
* SIGTERM is forwarded to tini so Kubernetes graceful shutdown still reaches
the Spark driver
* Only the {{driver}} entrypoint mode is affected; {{executor}} and
pass-through modes are unchanged
* The {{TERMINATION_LOG}} env var can override the path (e.g. for local
testing)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
