Zhanghao Chen created FLINK-30513:
-------------------------------------
Summary: HA storage dir leaks on cluster termination
Key: FLINK-30513
URL: https://issues.apache.org/jira/browse/FLINK-30513
Project: Flink
Issue Type: Improvement
Components: Runtime / Coordination
Affects Versions: 1.16.0, 1.15.0
Reporter: Zhanghao Chen
Attachments: image-2022-12-27-21-32-17-510.png
*Problem*
We found that HA storage dir leaks on cluster termination for a Flink job with
HA enabled. The following picture shows the HA storage dir (here on HDFS) of
the cluster czh-flink-test-offline (of application mode) after canelling the
job with flink-cancel. We are left with an empty dir, and too many empty dirs
will greatly hurt the stability of HDFS NameNode!
!image-2022-12-27-21-32-17-510.png|width=582,height=158!
Furthermore, in case the user choose to retain the checkpoints on job
termination, we will have the completedCheckpoints leaked as well. Note that we
no longer need the completedCheckpoints files as we'll directly recover
retained CPs from the CP data dir.
*Root Cause*
When we run AbstractHaServices#closeAndCleanupAllData(), we cleaned up blob
store, but didn't clean the HA storage dir.
*Proposal*
Clean up the HA storage dir after cleaning up blob store in
AbstractHaServices#closeAndCleanupAllData().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
