Peter Tian created SPARK-41946:
----------------------------------
Summary: DiskBlockManager can't clear all of the tmp dirs in k8s
Key: SPARK-41946
URL: https://issues.apache.org/jira/browse/SPARK-41946
Project: Spark
Issue Type: Bug
Components: Block Manager, Spark Core
Affects Versions: 3.3.1, 3.2.0
Environment: Spark 3.2.2
kubernetes v1.20.8
jdk1.8.0_162
Reporter: Peter Tian
I have a test for spark on k8s that my job had completed, but some tmp dir is
still remained. Those dirs are start with "blockmgr". My "spark.local.dirs" is
mount on local NVMe SSD.
The DiskBlockManager may be clear the dirs just one iteration ， then the
process be killed.
The shuffle data will be getting larger and fill the disk space.
!image-2023-01-09-14-03-01-840.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
