Attila Zsolt Piros created SPARK-37373: ------------------------------------------
Summary: Collect LocalSparkContext worker logs in case of test failure Key: SPARK-37373 URL: https://issues.apache.org/jira/browse/SPARK-37373 Project: Spark Issue Type: Improvement Components: Tests Affects Versions: 3.2.0 Reporter: Attila Zsolt Piros Assignee: Attila Zsolt Piros About 50 test suites are using LocalSparkContext by specifying "local-cluster" as the cluster URL. In this case executor logs will be under the worker dir which is a temp directory and as such will be deleted at shutdown (for details see https://github.com/apache/spark/blob/0a4961df29aab6912492e87e4e719865fe20d981/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala#L70) So when a test fails and the error was on the executor side the log will be lost. This is only for local cluster tests and not for standalone tests where logs will be kept in the "<SPARK_HOME>/work". -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org