[
https://issues.apache.org/jira/browse/IGNITE-19239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maksim Timonin reassigned IGNITE-19239:
---------------------------------------
Assignee: Maksim Timonin
> Checkpoint read lock acquisition timeouts during snapshot restore
> -----------------------------------------------------------------
>
> Key: IGNITE-19239
> URL: https://issues.apache.org/jira/browse/IGNITE-19239
> Project: Ignite
> Issue Type: Bug
> Reporter: Ilya Shishkov
> Assignee: Maksim Timonin
> Priority: Minor
> Labels: iep-43, ise
> Attachments: BlockingThreadsOnSnapshotRestoreReproducerTest.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There may be possible error messages about checkpoint read lock acquisition
> timeouts and critical threads blocking during snapshot restore process (just
> after caches start):
> {quote}
> [2023-04-06T10:55:46,561][ERROR]\[ttl-cleanup-worker-#475%node%][CheckpointTimeoutLock]
> Checkpoint read lock acquisition has been timed out.
> {quote}
> {quote}
> [2023-04-06T10:55:47,487][ERROR]\[tcp-disco-msg-worker-[crd]\-#23%node%\-#446%node%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour \[workerName=db-checkpoint-thread,
> threadName=db-checkpoint-thread-#457%snapshot.BlockingThreadsOnSnapshotRestoreReproducerTest0%,
> {color:red}blockedFor=100s{color}]
> {quote}
> Also there are active exchange process, which finishes with such timings
> (timing will be approximatelly equal to blocking time of threads):
> {quote}
> [2023-04-06T10:55:52,211][INFO
> ]\[exchange-worker-#450%node%][GridDhtPartitionsExchangeFuture] Exchange
> timings [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=5],
> resVer=AffinityTopologyVersion [topVer=1, minorTopVer=5], stage="Waiting in
> exchange queue" (0 ms), ..., stage="Restore partition states"
> ({color:red}100163 ms{color}), ..., stage="Total time" ({color:red}100334
> ms{color})]
> {quote}
>
> Most of time such errors and long lasting threads blocking tells that cluster
> is in emergency state or will crash very soon.
> So, there are two possible ways to solve problem:
> # If these errors do not affect restoring from snapshot and are false
> positive ones, they can confuse, so we should remove them from logs.
> # If these errors are not false positive, root cause of them have to be
> investigated and solved.
>
> How to reproduce:
> # Set checkpoint frequency less than failure detection timeout.
> # Ensure, that cache groups partitions states restoring lasts more than
> failure detection timeout, i.e. it is actual to sufficiently large caches.
> Reproducer: [^BlockingThreadsOnSnapshotRestoreReproducerTest.patch]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
