[
https://issues.apache.org/jira/browse/IGNITE-28042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denis Chudov updated IGNITE-28042:
----------------------------------
Description:
{code:java}
[2026-02-28T05:25:14,244][ERROR][%idnct_rdnrweznttazt_3345%metastorage-watch-executor-0][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeFailureHandler [nodeName=idnct_rdnrweznttazt_3345,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=CRITICAL_ERROR, failureCtxId=bca22f59-9242-486b-a1b2-0c5abfb5031f]
05:25:14 org.apache.ignite.internal.failure.StackTraceCapturingException:
Unknown error 05:25:14 at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:191)
05:25:14 at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:168)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyFailureHandlerOnFirstFailureInNotificationChain(WatchProcessor.java:446)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$3(WatchProcessor.java:240)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
05:25:14 at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
05:25:14 at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
05:25:14 at java.base/java.lang.Thread.run(Thread.java:833) 05:25:14
Caused by: java.util.concurrent.CompletionException: java.lang.AssertionError:
Causality token must be greater than the last completed: [token=85,
lastCompleted=85] 05:25:14 at
java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152)
05:25:14 ... 4 more 05:25:14 Caused by: java.lang.AssertionError:
Causality token must be greater than the last completed: [token=85,
lastCompleted=85] 05:25:14 at
org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:276)
05:25:14 at
org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:107)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:489)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:268)
05:25:14 at
org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:947)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:191)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:237)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
05:25:14 ... 4 more{code}
Logs were added in IGNITE-27125 - need to investigate.
was:
{code:java}
[2026-02-28T05:25:14,244][ERROR][%idnct_rdnrweznttazt_3345%metastorage-watch-executor-0][FailureManager]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=StopNodeFailureHandler [nodeName=idnct_rdnrweznttazt_3345,
super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=CRITICAL_ERROR, failureCtxId=bca22f59-9242-486b-a1b2-0c5abfb5031f]
05:25:14 org.apache.ignite.internal.failure.StackTraceCapturingException:
Unknown error 05:25:14 at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:191)
05:25:14 at
org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:168)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyFailureHandlerOnFirstFailureInNotificationChain(WatchProcessor.java:446)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$3(WatchProcessor.java:240)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
05:25:14 at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
05:25:14 at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
05:25:14 at java.base/java.lang.Thread.run(Thread.java:833) 05:25:14
Caused by: java.util.concurrent.CompletionException: java.lang.AssertionError:
Causality token must be greater than the last completed: [token=85,
lastCompleted=85] 05:25:14 at
java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152)
05:25:14 ... 4 more 05:25:14 Caused by: java.lang.AssertionError:
Causality token must be greater than the last completed: [token=85,
lastCompleted=85] 05:25:14 at
org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:276)
05:25:14 at
org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:107)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:489)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:268)
05:25:14 at
org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:947)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:191)
05:25:14 at
org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:237)
05:25:14 at
java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
05:25:14 ... 4 more{code}
> Causality token must be greater than the last completed (again)
> ---------------------------------------------------------------
>
> Key: IGNITE-28042
> URL: https://issues.apache.org/jira/browse/IGNITE-28042
> Project: Ignite
> Issue Type: Bug
> Reporter: Denis Chudov
> Priority: Major
> Labels: ignite-3
>
>
> {code:java}
>
> [2026-02-28T05:25:14,244][ERROR][%idnct_rdnrweznttazt_3345%metastorage-watch-executor-0][FailureManager]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=StopNodeFailureHandler [nodeName=idnct_rdnrweznttazt_3345,
> super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet
> [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
> failureCtx=CRITICAL_ERROR, failureCtxId=bca22f59-9242-486b-a1b2-0c5abfb5031f]
> 05:25:14 org.apache.ignite.internal.failure.StackTraceCapturingException:
> Unknown error 05:25:14 at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:191)
> 05:25:14 at
> org.apache.ignite.internal.failure.FailureManager.process(FailureManager.java:168)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyFailureHandlerOnFirstFailureInNotificationChain(WatchProcessor.java:446)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$3(WatchProcessor.java:240)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1163)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482)
> 05:25:14 at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
> 05:25:14 at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
> 05:25:14 at java.base/java.lang.Thread.run(Thread.java:833) 05:25:14
> Caused by: java.util.concurrent.CompletionException:
> java.lang.AssertionError: Causality token must be greater than the last
> completed: [token=85, lastCompleted=85] 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:377)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1152)
> 05:25:14 ... 4 more 05:25:14 Caused by: java.lang.AssertionError:
> Causality token must be greater than the last completed: [token=85,
> lastCompleted=85] 05:25:14 at
> org.apache.ignite.internal.causality.IncrementalVersionedValue.completeInternal(IncrementalVersionedValue.java:276)
> 05:25:14 at
> org.apache.ignite.internal.causality.IncrementalVersionedValue$1.onUpdate(IncrementalVersionedValue.java:107)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.notifyUpdateRevisionListeners(WatchProcessor.java:489)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$notifyWatchesInternal$7(WatchProcessor.java:268)
> 05:25:14 at
> org.apache.ignite.internal.util.IgniteUtils.inBusyLockAsync(IgniteUtils.java:947)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.inBusyLockAsync(WatchProcessor.java:191)
> 05:25:14 at
> org.apache.ignite.internal.metastorage.server.WatchProcessor.lambda$enqueue$2(WatchProcessor.java:237)
> 05:25:14 at
> java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
> 05:25:14 ... 4 more{code}
>
> Logs were added in IGNITE-27125 - need to investigate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
