[
https://issues.apache.org/jira/browse/IGNITE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor updated IGNITE-22327:
--------------------------
Description:
*Steps to reproduce:*
# Start cluster of 3 nodes on 3 hosts.
# Create 10 tables and insert 10 rows into each.
# Kill 1 server.
# Start the killed server.
# Check logs for errors.
*Expected:*
No errors in logs.
*Actual:*
Errors in logs
{code:java}
2024-05-23 21:09:52:473 +0300
[ERROR][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][StateMachineAdapter]
Encountered an error=Status[ESTATEMACHINE<10002>: StateMachine meet critical
error when applying one or more tasks since index=2,
Status[ESTATEMACHINE<10002>: No serializer provider defined for group type 40
and message type 8]] on StateMachine
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine,
it's highly recommended to implement this method as raft stops working since
some error occurs, you should figure out the cause and repair or remove this
node.
Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
StateMachine meet critical error when applying one or more tasks since index=2,
Status[ESTATEMACHINE<10002>: No serializer provider defined for group type 40
and message type 8]]]
at
org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:156)
at
org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:147)
at
org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:72)
at
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:803)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:557)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:525)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:444)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:136)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:130)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:340)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:278)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:842)
2024-05-23 21:09:52:473 +0300
[WARNING][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][NodeImpl]
Node <18_part_19/ClusterFailover3NodesTest_cluster_0> got error: Error
[type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
StateMachine meet critical error when applying one or more tasks since index=2,
Status[ESTATEMACHINE<10002>: No serializer provider defined for group type 40
and message type 8]]].
2024-05-23 21:09:52:473 +0300
[WARNING][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][FSMCallerImpl]
FSMCaller already in error status, ignore new error
Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
StateMachine meet critical error when applying one or more tasks since index=2,
Status[ESTATEMACHINE<10002>: No serializer provider defined for group type 40
and message type 8]]]
at
org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:156)
at
org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:147)
at
org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:72)
at
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:803)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:557)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:525)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:444)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:136)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:130)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:340)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:278)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:842){code}
was:
*Steps to reproduce:*
# Start cluster of 3 nodes on 3 hosts.
# Create 10 tables and insert 10 rows into each.
# Kill 1 server.
# Start the killed server.
# Check logs for errors.
*Expected:*
No errors in logs.
*Actual:*
Errors in logs
{code:java}
2024-05-17 04:26:37:808 +0000
[ERROR][%ClusterFailover2NodesTest_cluster_0%common-scheduler-0][CriticalWorkerWatchdog]
A critical thread is blocked for 688 ms that is more than the allowed 500 ms,
it is "ClusterFailover2NodesTest_cluster_0-srv-worker-3" prio=10 Id=41 RUNNABLE
at
app//org.apache.ignite.internal.network.message.InvokeResponseDeserializer.getMessage(InvokeResponseDeserializer.java:25)
at
app//org.apache.ignite.internal.network.message.InvokeResponseDeserializer.getMessage(InvokeResponseDeserializer.java:11)
at
app//org.apache.ignite.internal.network.netty.InboundDecoder.decode(InboundDecoder.java:136)
at
app//io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
at
app//io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
at
app//io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
at
app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at
app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
app//io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at
app//io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at
app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at
app//io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at
app//io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at
app//io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at
app//io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at
app//io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at
app//io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at
app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at
app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at [email protected]/java.lang.Thread.run(Thread.java:833) {code}
GC calls of node ClusterFailover2NodesTest_cluster_0 (LOG: [^ignite3db-0.log])
!image-2024-05-17-18-06-23-594.png! GC calls of node
ClusterFailover2NodesTest_cluster_1 (LOG: [^ignite3db-0.log])
!image-2024-05-17-18-06-04-081.png!
> Error "StateMachine meet critical error" on restart
> ----------------------------------------------------
>
> Key: IGNITE-22327
> URL: https://issues.apache.org/jira/browse/IGNITE-22327
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 3.0.0-beta2
> Environment: 3 nodes (with arguments "-Xms4096m", "-Xmx4096m" ) on *3
> hosts*
> Reporter: Igor
> Priority: Major
> Labels: ignite-3
>
> *Steps to reproduce:*
> # Start cluster of 3 nodes on 3 hosts.
> # Create 10 tables and insert 10 rows into each.
> # Kill 1 server.
> # Start the killed server.
> # Check logs for errors.
> *Expected:*
> No errors in logs.
> *Actual:*
> Errors in logs
> {code:java}
> 2024-05-23 21:09:52:473 +0300
> [ERROR][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][StateMachineAdapter]
> Encountered an error=Status[ESTATEMACHINE<10002>: StateMachine meet critical
> error when applying one or more tasks since index=2,
> Status[ESTATEMACHINE<10002>: No serializer provider defined for group type 40
> and message type 8]] on StateMachine
> org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine,
> it's highly recommended to implement this method as raft stops working since
> some error occurs, you should figure out the cause and repair or remove this
> node.
> Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
> StateMachine meet critical error when applying one or more tasks since
> index=2, Status[ESTATEMACHINE<10002>: No serializer provider defined for
> group type 40 and message type 8]]]
> at
> org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:156)
> at
> org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:147)
> at
> org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:72)
> at
> org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:803)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:557)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:525)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:444)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:136)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:130)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:340)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:278)
> at
> com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:842)
> 2024-05-23 21:09:52:473 +0300
> [WARNING][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][NodeImpl]
> Node <18_part_19/ClusterFailover3NodesTest_cluster_0> got error: Error
> [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
> StateMachine meet critical error when applying one or more tasks since
> index=2, Status[ESTATEMACHINE<10002>: No serializer provider defined for
> group type 40 and message type 8]]].
> 2024-05-23 21:09:52:473 +0300
> [WARNING][%ClusterFailover3NodesTest_cluster_0%JRaft-FSMCaller-Disruptor_stripe_3-0][FSMCallerImpl]
> FSMCaller already in error status, ignore new error
> Error [type=ERROR_TYPE_STATE_MACHINE, status=Status[ESTATEMACHINE<10002>:
> StateMachine meet critical error when applying one or more tasks since
> index=2, Status[ESTATEMACHINE<10002>: No serializer provider defined for
> group type 40 and message type 8]]]
> at
> org.apache.ignite.raft.jraft.core.IteratorImpl.getOrCreateError(IteratorImpl.java:156)
> at
> org.apache.ignite.raft.jraft.core.IteratorImpl.setErrorAndRollback(IteratorImpl.java:147)
> at
> org.apache.ignite.raft.jraft.core.IteratorWrapper.setErrorAndRollback(IteratorWrapper.java:72)
> at
> org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:803)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:557)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:525)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:444)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:136)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:130)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:340)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:278)
> at
> com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:842){code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
