Takuya Ueshin created SPARK-36062:
-------------------------------------
Summary: Try to capture faulthanlder when a Python worker crashes.
Key: SPARK-36062
URL: https://issues.apache.org/jira/browse/SPARK-36062
Project: Spark
Issue Type: Improvement
Components: PySpark
Affects Versions: 3.2.0
Reporter: Takuya Ueshin
Currently, we just see an error message saying "exited unexpectedly (crashed)"
when the UDFs causes the Python worker to crash by like segmentation fault.
We should take advantage of
{{[faulthandler|https://docs.python.org/3/library/faulthandler.html]}} and try
to capture the error message from the {{faulthandler}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
