Jiang Xingbo created SPARK-24340: ------------------------------------ Summary: Clean up non-shuffle disk block manager files following executor death Key: SPARK-24340 URL: https://issues.apache.org/jira/browse/SPARK-24340 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.0 Reporter: Jiang Xingbo
Currently we only clean up local folders on application removed, and we don't clean up non-shuffle files, such as temp. shuffle blocks, cached RDD/broadcast blocks, spill files, etc. and this can cause disk space leaks when executors periodically die and are replaced. To avoid this source of disk space leak, we can clean up executor disk store files except for shuffle index and data files on executor finished. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org