Xingbo Jiang created SPARK-36500:
------------------------------------
Summary: The temp_shuffle files are not cleaned up when a task is
interrupted
Key: SPARK-36500
URL: https://issues.apache.org/jira/browse/SPARK-36500
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.1.2, 3.0.3, 2.4.8
Reporter: Xingbo Jiang
Assignee: Xingbo Jiang
We discovered a potential file leak on the long running cluster. As the cluster
running for a few hours, there are many "temp_shuffle_*" files left in the
folder. It seems the files are not cleaned up when the task finishes as
expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
