Attila Zsolt Piros created SPARK-37373:
------------------------------------------
Summary: Collect LocalSparkContext worker logs in case of test
failure
Key: SPARK-37373
URL: https://issues.apache.org/jira/browse/SPARK-37373
Project: Spark
Issue Type: Improvement
Components: Tests
Affects Versions: 3.2.0
Reporter: Attila Zsolt Piros
Assignee: Attila Zsolt Piros
About 50 test suites are using LocalSparkContext by specifying "local-cluster"
as the cluster URL. In this case executor logs will be under the worker dir
which is a temp directory and as such will be deleted at shutdown (for details
see
https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70)
So when a test fails and the error was on the executor side the log will be
lost.
This is only for local cluster tests and not for standalone tests where logs
will be kept in the "<SPARK_HOME>/work".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
