[
https://issues.apache.org/jira/browse/IGNITE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vyacheslav Koptilin updated IGNITE-16423:
-----------------------------------------
Labels: ignite-3 (was: )
> [%my-first-node%JRaft-Common-Executor-1][SnapshotExecutorImpl] Fail to close
> writer
> -----------------------------------------------------------------------------------
>
> Key: IGNITE-16423
> URL: https://issues.apache.org/jira/browse/IGNITE-16423
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha3
> Reporter: Fedor Malchikov
> Priority: Critical
> Labels: ignite-3
> Attachments: my-first-node(sql).log, my-first-node.log
>
>
> {code:java}
> 2022-01-28 13:17:41:192 +0300
> [ERROR][%my-first-node%JRaft-Common-Executor-1][SnapshotExecutorImpl] Fail to
> close writerjava.io.IOException at
> org.apache.ignite.raft.jraft.storage.snapshot.local.LocalSnapshotStorage.close(LocalSnapshotStorage.java:242)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.local.LocalSnapshotWriter.close(LocalSnapshotWriter.java:93)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.local.LocalSnapshotWriter.close(LocalSnapshotWriter.java:88)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl.onSnapshotSaveDone(SnapshotExecutorImpl.java:387)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.continueRun(SnapshotExecutorImpl.java:135)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.lambda$run$0(SnapshotExecutorImpl.java:131)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)2022-01-28 13:17:41:202
> +0300
> [ERROR][%my-first-node%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter]
> Encountered an error=Status[EIO<1014>: Fail to save snapshot.] on
> StateMachine
> org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine,
> it's highly recommended to implement this method as raft stops working since
> some error occurs, you should figure out the cause and repair or remove this
> node.Error [type=ERROR_TYPE_SNAPSHOT, status=Status[EIO<1014>: Fail to save
> snapshot.]] at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl.reportError(SnapshotExecutorImpl.java:682)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl.onSnapshotSaveDone(SnapshotExecutorImpl.java:406)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.continueRun(SnapshotExecutorImpl.java:135)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.lambda$run$0(SnapshotExecutorImpl.java:131)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)2022-01-28 13:17:41:203
> +0300
> [WARNING][%my-first-node%JRaft-FSMCaller-Disruptor-_stripe_10-0][NodeImpl]
> Node <metastorage_raft_group/127.0.1.1:3344> got error: Error
> [type=ERROR_TYPE_SNAPSHOT, status=Status[EIO<1014>: Fail to save
> snapshot.]].2022-01-28 13:17:41:203 +0300
> [WARNING][%my-first-node%JRaft-FSMCaller-Disruptor-_stripe_10-0][FSMCallerImpl]
> FSMCaller already in error status, ignore new errorError
> [type=ERROR_TYPE_SNAPSHOT, status=Status[EIO<1014>: Fail to save snapshot.]]
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl.reportError(SnapshotExecutorImpl.java:682)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl.onSnapshotSaveDone(SnapshotExecutorImpl.java:406)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.continueRun(SnapshotExecutorImpl.java:135)
> at
> org.apache.ignite.raft.jraft.storage.snapshot.SnapshotExecutorImpl$SaveSnapshotDone.lambda$run$0(SnapshotExecutorImpl.java:131)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)2022-01-28 13:17:41:205
> +0300
> [INFO][%my-first-node%JRaft-FSMCaller-Disruptor-_stripe_10-0][ReplicatorGroupImpl]
> Fail to find the next candidate.2022-01-28 13:17:41:205 +0300
> [INFO][%my-first-node%JRaft-FSMCaller-Disruptor-_stripe_10-0][StateMachineAdapter]
> onLeaderStop: status=Status[EBADNODE<10009>: Raft node(leader or candidate)
> is in error.].2022-01-28 13:17:51:262 +0300
> [ERROR][Thread-72][MetaStorageServiceImpl] Unexpected exceptionclass
> org.apache.ignite.lang.IgniteInternalException:
> java.util.concurrent.TimeoutException at
> org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:121)
> at
> org.apache.ignite.internal.metastorage.client.MetaStorageServiceImpl$WatchProcessor$Watcher.run(MetaStorageServiceImpl.java:476)Caused
> by: java.util.concurrent.ExecutionException:
> java.util.concurrent.TimeoutException at
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> at
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
> at
> org.apache.ignite.internal.metastorage.client.CursorImpl$InnerIterator.hasNext(CursorImpl.java:113)
> ... 1 moreCaused by: java.util.concurrent.TimeoutException at
> org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl.sendWithRetry(RaftGroupServiceImpl.java:502)
> at
> org.apache.ignite.raft.jraft.rpc.impl.RaftGroupServiceImpl$1.lambda$accept$2(RaftGroupServiceImpl.java:555)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> And after that, the timeout error repeats every 10 seconds.
> The problem is reproducible stably, but each time needs a different time. In
> attachment 2 different test: one without any activity and one houre waite ,
> second after sql table creation
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
