[
https://issues.apache.org/jira/browse/SPARK-37373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Zsolt Piros updated SPARK-37373:
---------------------------------------
Affects Version/s: 3.3.0
(was: 3.2.0)
> Collect LocalSparkContext worker logs in case of test failure
> -------------------------------------------------------------
>
> Key: SPARK-37373
> URL: https://issues.apache.org/jira/browse/SPARK-37373
> Project: Spark
> Issue Type: Improvement
> Components: Tests
> Affects Versions: 3.3.0
> Reporter: Attila Zsolt Piros
> Assignee: Attila Zsolt Piros
> Priority: Major
>
> About 50 test suites are using LocalSparkContext by specifying
> "local-cluster" as the cluster URL. In this case executor logs will be under
> the worker dir which is a temp directory and as such will be deleted at
> shutdown (for details see
> https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70)
> So when a test fails and the error was on the executor side the log will be
> lost.
> This is only for local cluster tests and not for standalone tests where logs
> will be kept in the "<SPARK_HOME>/work".
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
