[
https://issues.apache.org/jira/browse/FLINK-13856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325089#comment-17325089
]
Stephan Ewen edited comment on FLINK-13856 at 4/19/21, 2:51 PM:
----------------------------------------------------------------
I think this is an interesting one to reactivate in the next release cycle.
At the moment, most committers are pretty busy testing 1.13 and putting in the
last stabilization fixes.
But I see the point for this, so let's look at this in the next cycle.
I learned a bit more about S3 and I think there are some optimizations in the
S3 connectors that make recursive deletes not {{O(1)}}, but {{O(n)}} with a
smaller {{O}}, which is already a good win.
My biggest concern right now is that we need to find a good way to internally
separate the discard of shared state handles from exclusive shared handles.
Otherwise, if we naively delete the exclusive directory and then dispose all
state handles (to be sure to remove shared state), we still ping the FS /
Namenode once per each state file (most of which files don't exist any more).
That we need to avoid.
was (Author: stephanewen):
I think this is an interesting one to reactivate in the next release cycle.
At the moment, most committers are pretty busy testing 1.13 and putting in the
last stabilization fixes.
But I see the point for this, so let's look at this in the next cycle.
I learned a bit more about S3 and I think there are some optimizations in the
S3 connectors that make recursive deletes not O(1), but O(n) with a smaller O,
which is already a good win.
My biggest concern right now is that we need to find a good way to internally
separate the discard of shared state handles from exclusive shared handles.
Otherwise, if we naively delete the exclusive directory and then dispose all
state handles (to be sure to remove shared state), we still ping the FS /
Namenode once per each state file (most of which files don't exist any more).
That we need to avoid.
> Reduce the delete file api when the checkpoint is completed
> -----------------------------------------------------------
>
> Key: FLINK-13856
> URL: https://issues.apache.org/jira/browse/FLINK-13856
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / State Backends
> Affects Versions: 1.8.1, 1.9.0
> Reporter: Andrew.D.lin
> Assignee: Andrew.D.lin
> Priority: Major
> Labels: pull-request-available, stale-assigned
> Attachments: after.png, before.png,
> f6cc56b7-2c74-4f4b-bb6a-476d28a22096.png
>
> Original Estimate: 48h
> Time Spent: 10m
> Remaining Estimate: 47h 50m
>
> When the new checkpoint is completed, an old checkpoint will be deleted by
> calling CompletedCheckpoint.discardOnSubsume().
> When deleting old checkpoints, follow these steps:
> 1, drop the metadata
> 2, discard private state objects
> 3, discard location as a whole
> In some cases, is it possible to delete the checkpoint folder recursively by
> one call?
> As far as I know the full amount of checkpoint, it should be possible to
> delete the folder directly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
