[ https://issues.apache.org/jira/browse/FLINK-33288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776183#comment-17776183 ]
Zhanghao Chen commented on FLINK-33288: --------------------------------------- HI [~mapohl], thanks for joining the discussion here. +1 for adding recursive deletion of empty folders within the HA storage folder + the JRS storage folder. Actually, only one-level of recursion should be enough in the two cases here. For HA storage folder cleanup, we just need to add an additional check if there exist some other files/subfolders before delete the dir based on FLINK-30513's solution. For cleaning the JRS storage folder, we'll need to implement a cleanup method for JobResultStore, which is called on cluster termination and takes responsibility of cleaning up any empty folders. I'd be willing to pick up from where I left in FLINK-30513 to fix the leakage issue. > Empty directory residue with appid name in HA(highly-available) related > directory of hdfs, not cleaned > ------------------------------------------------------------------------------------------------------ > > Key: FLINK-33288 > URL: https://issues.apache.org/jira/browse/FLINK-33288 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration > Affects Versions: 1.16.2, 1.17.1 > Reporter: Xin Chen > Priority: Major > Attachments: image-2023-10-17-16-43-07-859.png, screenshot-1.png, > screenshot-2.png > > > When I submitted a large number of tasks in Flink-on-Yarn mode and > successfully executed, I unexpectedly found a large number of empty > directories left in the directory related to 'high availability.storageDir' > on hdfs, with appids as shown below. I believe this must be cleared! However, > after verification in the environments of 1.16.2 and 1.17.1, it was proven > that neither of them solved this problem. > my flink-conf.yaml about 'high availability.storageDir': > {code:java} > high-availability.storageDir: hdfs://hdfsHACluster/flink/recovery > {code} > !screenshot-1.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)