Lantao Jin created SPARK-30069:
----------------------------------
Summary: Clean up non-shuffle disk block manager files following
executor exists on YARN
Key: SPARK-30069
URL: https://issues.apache.org/jira/browse/SPARK-30069
Project: Spark
Issue Type: Bug
Components: YARN
Affects Versions: 3.0.0
Reporter: Lantao Jin
Currently we only clean up the local directories on application removed.
However, when executors die and restart repeatedly, many temp files are left
untouched in the local directories, which is undesired behavior and could cause
disk space used up gradually.
SPARK-24340 had fixed this problem in standalone mode. But in YARN mode, this
issue still exists. Especially, in long running service like Spark
thrift-server with dynamic resource allocation disabled, it's very easy causes
local disk full.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
