Holden Karau created SPARK-49788:
------------------------------------
Summary: Add spark.cleaner.ttl functionality for long lived jobs
Key: SPARK-49788
URL: https://issues.apache.org/jira/browse/SPARK-49788
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.0
Reporter: Holden Karau
Assignee: Holden Karau
We should add a TTL (and maybe a threshold?) to clean shuffle files which have
been stored for greater than some fixed period of time. This would be useful
for long lived jobs with Spark Connect and Notebooks where items may not go
out-of-scope and be resident for longer than needed.
See SPARK-7689 which removed the original TTL cleaner.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
