[
https://issues.apache.org/jira/browse/FLINK-30513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17675967#comment-17675967
]
Yang Wang commented on FLINK-30513:
-----------------------------------
Given that the cluster-id is configured to k8s-app-ha-1-116, we will have
following files in the HA storage directory. The completedCheckpoint and job
graph files will be deleted once the job reached global terminal state(e.g.
CANCELED, FAILED, FINISHED). So currently we only clean up the blob directory
in the {{{}HighAvailabilityServices#closeAndCleanupAllData{}}}. And then the
parent directory will be residual.
It is not a serious problem when using cloud object storage(S3, OSS, etc.).
However, it is critical for HDFS since leaked directories will consume too much
memory of NameNode.
I am +1 to delete the parent directory in
{{HighAvailabilityServices#closeAndCleanupAllData.}}
{code:java}
wangyang-pc:scripts danrtsey.wy$ ossutil ls oss://flink-test-yiqi/flink-ha/
LastModifiedTime Size(B) StorageClass ETAG
ObjectName
2023-01-12 19:42:32 +0800 CST 0 Standard
D41D8CD98F00B204E9800998ECF8427E
oss://flink-test-yiqi/flink-ha/job-result-store/k8s-app-ha-1-116/
2023-01-12 19:44:16 +0800 CST 0 Standard
D41D8CD98F00B204E9800998ECF8427E
oss://flink-test-yiqi/flink-ha/k8s-app-ha-1-116/
2023-01-12 19:42:32 +0800 CST 0 Standard
D41D8CD98F00B204E9800998ECF8427E
oss://flink-test-yiqi/flink-ha/k8s-app-ha-1-116/blob/
2023-01-12 19:42:44 +0800 CST 5426525 Standard
EF151FCD3D1F91C3EB512118F05D2E20
oss://flink-test-yiqi/flink-ha/k8s-app-ha-1-116/blob/job_ffffffffe07004590000000000000000/blob_p-a28c3727cd8d501e6b024187013e4311079499be-b327aba6b7fee1f6b0fb4fe66a88a637
2023-01-12 19:44:16 +0800 CST 14288 Standard
0983C3E02FBC7A904207A12295A3AF28
oss://flink-test-yiqi/flink-ha/k8s-app-ha-1-116/completedCheckpoint4679f587e15a
2023-01-12 19:42:44 +0800 CST 34080 Standard
B37673CA07B30B8A0FC76939BBDFA9F7
oss://flink-test-yiqi/flink-ha/k8s-app-ha-1-116/submittedJobGraph427419c216b4
{code}
> HA storage dir leaks on cluster termination
> --------------------------------------------
>
> Key: FLINK-30513
> URL: https://issues.apache.org/jira/browse/FLINK-30513
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.15.0, 1.16.0
> Reporter: Zhanghao Chen
> Priority: Major
> Attachments: image-2022-12-27-21-32-17-510.png
>
>
> *Problem*
> We found that HA storage dir leaks on cluster termination for a Flink job
> with HA enabled. The following picture shows the HA storage dir (here on
> HDFS) of the cluster czh-flink-test-offline (of application mode) after
> canelling the job with flink-cancel. We are left with an empty dir, and too
> many empty dirs will greatly hurt the stability of HDFS NameNode!
> !image-2022-12-27-21-32-17-510.png|width=582,height=158!
>
> Furthermore, in case the user choose to retain the checkpoints on job
> termination, we will have the completedCheckpoints leaked as well. Note that
> we no longer need the completedCheckpoints files as we'll directly recover
> retained CPs from the CP data dir.
> *Root Cause*
> When we run AbstractHaServices#closeAndCleanupAllData(), we cleaned up blob
> store, but didn't clean the HA storage dir.
> *Proposal*
> Clean up the HA storage dir after cleaning up blob store in
> AbstractHaServices#closeAndCleanupAllData().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
