GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/21390
[SPARK-24340][Core] Clean up non-shuffle disk block manager files following
executor death
## What changes were proposed in this pull request?
Currently we only clean up the local directories on application removed.
However, when executors die and restart repeatedly, many temp files are left
untouched in the local directories, which is undesired behavior and could cause
disk space used up gradually.
We can detect executor death in the Worker, and clean up the non-shuffle
files (files not ended with ".index" or ".data") in the local directories, we
should not touch the shuffle files since they are expected to be used by the
external shuffle service.
## How was this patch tested?
Add new test suite to cover.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark cleanupNonshuffleFiles
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21390.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21390
----
commit 251726ec66805804c12d5ed9caf4fa84bfe8a298
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-05-08T15:29:03Z
implement delete non-shuffle files on executor finished.
commit 2ba90a237bbb013d913eb8aceffe41ee49ec1ca0
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-05-10T17:26:45Z
refactor
commit 64bde5f43a3a4e64f8ce5d69f03997ca10508431
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-05-21T13:16:07Z
add test suite
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
