[
https://issues.apache.org/jira/browse/FLINK-33090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17769854#comment-17769854
]
Yi Zhang commented on FLINK-33090:
----------------------------------
some of our users jobs having large number of state files, this is an annoying
issue for them, really appreciate if can resolve this.
[@rkhachatryan|https://github.com/rkhachatryan] can you kindly help review as
requested? thank you!
> CheckpointsCleaner clean individual checkpoint states in parallel
> -----------------------------------------------------------------
>
> Key: FLINK-33090
> URL: https://issues.apache.org/jira/browse/FLINK-33090
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.17.1
> Reporter: Yi Zhang
> Assignee: Yi Zhang
> Priority: Major
> Labels: pull-request-available
>
> Currently CheckpointsCleaner clean multiple checkpoints in parallel with
> JobManager's ioExecutor, however each checkpoint states is cleaned
> sequentially. With thousands of StateObjects to clean this can take long time
> on some checkpoint storage, if longer than the checkpoint interval this
> prevents new checkpointing.
> The proposal is to use the same ioExecutor to clean up each checkpoints
> states in parallel as well. From my local testing, with default settings for
> ioExecutor thread pool for xK state files this can reduce clean up time from
> 10 minutes to <1 minute.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
