[
https://issues.apache.org/jira/browse/FLINK-24086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17408090#comment-17408090
]
Piotr Nowojski commented on FLINK-24086:
----------------------------------------
Thanks for reporting this. Can you share what code path is taking long time?
And also what exactly is taking such a long time, ideally a profiler log? Do
you mean when restarting a single subtask/task from the job? Or restarting
whole job while keeping JM/CheckpointCoordinator running? I'm also not sure if
you have seen this comment in
{{CheckpointCoordinator#restoreLatestCheckpointedStateInternal}}:
{code:java}
// We create a new shared state registry object, so that all
pending async disposal
// requests from previous runs will go against the old object (were
they can do no
// harm). This must happen under the checkpoint lock.
sharedStateRegistry.close();
sharedStateRegistry = sharedStateRegistryFactory.create(executor);
{code}
that was introduced as a result of FLINK-7268.
> Do not re-register SharedStateRegistry to reduce the recovery time of the job
> -----------------------------------------------------------------------------
>
> Key: FLINK-24086
> URL: https://issues.apache.org/jira/browse/FLINK-24086
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Reporter: ming li
> Priority: Major
>
> At present, we only recover the {{CompletedCheckpointStore}} when the
> {{JobManager}} starts, so it seems that we do not need to re-register the
> {{SharedStateRegistry}} when the task restarts.
> The reason for this issue is that in our production environment, we discard
> part of the data and state to only restart the failed task, but found that it
> may take several seconds to register the {{SharedStateRegistry}} (thousands
> of tasks and dozens of TB states). When there are a large number of task
> failures at the same time, this may take several minutes (number of tasks *
> several seconds).
> Therefore, if the {{SharedStateRegistry}} can be reused, the time for task
> recovery can be reduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
