JAkutenshi commented on PR #6389:
URL: https://github.com/apache/ignite-3/pull/6389#issuecomment-3183114904
How I checked logging behavior:
1. Throw in `PartitionListener` `ReplicationTimeoutException` (doesn't
require the handling):
```
} else if (command instanceof SafeTimeSyncCommand) {
result = handleSafeTimeSyncCommand((SafeTimeSyncCommand)
command, commandIndex, commandTerm);
throw new ReplicationTimeoutException(null);
}
```
2. Run any truly integration long running test like
`ItRebalanceDistributedTest#testRebalanceWithTheSameNodes`.
3. In the log we should see at least one (but not a lot) message:
```
[2025-08-13T15:59:09,928][WARN
][%irdt_trwtsn_20002%Raft-Group-Client-5][ReplicaManager] Failed to sync safe
time for partition, the same kind of issue may affect all other replicas on
this node [groupId=3_part_0, attempt=10].
java.util.concurrent.CompletionException:
java.util.concurrent.TimeoutException: Send with retry timed out [retryCount =
50, groupId = 3_part_0, traceId = null, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.replicator.command.SafeTimeSyncCommandImpl),
originCommand = null, retryReasons = [[time=1755079144912, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:04,912], [time=1755079145112, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=199, attemptDuration=1,
attemptStartTime=2025-08-13T15:59:05,112], [time=1755079145313, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=199, attemptDuration=2,
attemptStartTime=2025-08-13T15:59:05,313], [time=1755079145513, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4, attem
ptStartTime=2025-08-13T15:59:05,513], [time=1755079145714, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:05,714], [time=1755079145914, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:05,914], [time=1755079146115, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:06,115], [time=1755079146315, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:06,315], [time=1755079146515, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:06,515], [time=1755079146716, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.; at
temptWaitDuration=196, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:06,716], [time=1755079146916, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:06,916], [time=1755079147116, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:07,116], [time=1755079147317, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:07,317], [time=1755079147519, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=194, attemptDuration=8,
attemptStartTime=2025-08-13T15:59:07,519], [time=1755079147720, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=192, attemptDuration=9,
attemptStartTime=2025-08-13T15:59:07,720], [time=1755079147921, msg=Peer
irdt_trwtsn_
20002:0 returned code EPERM: Is not leader.; attemptWaitDuration=199,
attemptDuration=2, attemptStartTime=2025-08-13T15:59:07,921],
[time=1755079148122, msg=Peer irdt_trwtsn_20000:0 returned code EPERM: Is not
leader.; attemptWaitDuration=199, attemptDuration=2,
attemptStartTime=2025-08-13T15:59:08,122], [time=1755079148323, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=198, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:08,323], [time=1755079148523, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:08,523], [time=1755079148724, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:08,724], [time=1755079148925, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:08,9
25], [time=1755079149125, msg=Peer irdt_trwtsn_20002:0 returned code EPERM: Is
not leader.; attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:09,125], [time=1755079149326, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:09,326], [time=1755079149526, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:09,526], [time=1755079149727, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:09,727]], stopTime = 1755079149889,
currentTime = 1755079149927, startTime = 1755079139889, duration = 10038].
at
java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
at
java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
at
java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632)
at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at
java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:686)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:660)
at
org.apache.ignite.internal.raft.RaftGroupServiceImpl.lambda$scheduleRetry$51(RaftGroupServiceImpl.java:910)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.util.concurrent.TimeoutException: Send with retry timed out
[retryCount = 50, groupId = 3_part_0, traceId = null, request =
org.apache.ignite.raft.jraft.rpc.WriteActionRequestImpl(org.apache.ignite.internal.replicator.command.SafeTimeSyncCommandImpl),
originCommand = null, retryReasons = [[time=1755079144912, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:04,912], [time=1755079145112, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=199, attemptDuration=1,
attemptStartTime=2025-08-13T15:59:05,112], [time=1755079145313, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=199, attemptDuration=2,
attemptStartTime=2025-08-13T15:59:05,313], [time=1755079145513, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4, attemptStartTime=2025-08-13T15:59:05
,513], [time=1755079145714, msg=Peer irdt_trwtsn_20002:0 returned code EPERM:
Is not leader.; attemptWaitDuration=197, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:05,714], [time=1755079145914, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:05,914], [time=1755079146115, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:06,115], [time=1755079146315, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:06,315], [time=1755079146515, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:06,515], [time=1755079146716, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptD
uration=5, attemptStartTime=2025-08-13T15:59:06,716], [time=1755079146916,
msg=Peer irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:06,916], [time=1755079147116, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:07,116], [time=1755079147317, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:07,317], [time=1755079147519, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=194, attemptDuration=8,
attemptStartTime=2025-08-13T15:59:07,519], [time=1755079147720, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=192, attemptDuration=9,
attemptStartTime=2025-08-13T15:59:07,720], [time=1755079147921, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is
not leader.; attemptWaitDuration=199, attemptDuration=2,
attemptStartTime=2025-08-13T15:59:07,921], [time=1755079148122, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=199, attemptDuration=2,
attemptStartTime=2025-08-13T15:59:08,122], [time=1755079148323, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=198, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:08,323], [time=1755079148523, msg=Peer
irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=3,
attemptStartTime=2025-08-13T15:59:08,523], [time=1755079148724, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:08,724], [time=1755079148925, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=197, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:08,925], [time=1755079149125, msg=P
eer irdt_trwtsn_20002:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=4,
attemptStartTime=2025-08-13T15:59:09,125], [time=1755079149326, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=196, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:09,326], [time=1755079149526, msg=Peer
irdt_trwtsn_20000:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=5,
attemptStartTime=2025-08-13T15:59:09,526], [time=1755079149727, msg=Peer
irdt_trwtsn_20001:0 returned code EPERM: Is not leader.;
attemptWaitDuration=195, attemptDuration=6,
attemptStartTime=2025-08-13T15:59:09,727]], stopTime = 1755079149889,
currentTime = 1755079149927, startTime = 1755079139889, duration = 10038].
at
org.apache.ignite.internal.raft.RetryContext.createTimeoutException(RetryContext.java:206)
... 9 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscr...@ignite.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
