[
https://issues.apache.org/jira/browse/IGNITE-12456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Kumar Powli updated IGNITE-12456:
--------------------------------------
Description:
We have Apache Ignite 3 node cluster setup with Amazon S3 Based Discovery.we
are into AWS cloud environment with Microservice model with 8 microservices.we
are using Ignite for session data store.While performing load test we are
facing data grid issues if the number of clients reached 40 , Once data grid
got corrupted we will lost the session store data and Application will not
respond due to session data also corrupted and all the instances will auto
scaled down to initial size 8.We need to restart Apache Ignite to make the
Application up.Please find the attached Apache Ignite Configuration for your
reference.
Apache Ignite Version : 2.7.0
07:24:46,678][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G] Blocked
system-critical thread has been detected. This can lead to cluster-wide
undefined behaviour [threadName=sys-stripe-5,
blockedFor=21s]07:24:46,678][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-5,
blockedFor=21s][07:24:46,680][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465499]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465499] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:52,692][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=ttl-cleanup-worker,
blockedFor=27s][07:24:52,692][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=DataStoreIgniteCache,
finished=false, heartbeatTs=1575271465044]]]class
org.apache.ignite.IgniteException: GridWorker [name=ttl-cleanup-worker,
igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465044] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:55,694][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-1,
blockedFor=24s][07:24:55,694][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-1, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271471538]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-1, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271471538] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:58,700][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-4,
blockedFor=29s][07:24:58,700][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-4, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271469397]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-4, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271469397] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:59,833][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-7,
blockedFor=29s][07:24:59,833][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-7, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271469973]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-7, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271469973] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:04,716][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-0,
blockedFor=39s][07:25:04,716][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-0, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271464782]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-0, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271464782] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:05,870][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-3,
blockedFor=34s][07:25:05,870][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-3, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271471528]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-3, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271471528] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:07,011][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-6,
blockedFor=41s][07:25:07,011][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-6, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465428]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-6, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465428] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:08,167][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-5,
blockedFor=42s][07:25:08,167][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465499]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465499] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:09,310][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=sys-stripe-2,
blockedFor=40s][07:25:09,311][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=sys-stripe-2, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271468397]]]class org.apache.ignite.IgniteException: GridWorker
[name=sys-stripe-2, igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271468397] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:14,183][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
Blocked system-critical thread has been detected. This can lead to
cluster-wide undefined behaviour [threadName=ttl-cleanup-worker,
blockedFor=49s][07:25:14,183][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
[ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
[type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
[name=ttl-cleanup-worker, igniteInstanceName=DataStoreIgniteCache,
finished=false, heartbeatTs=1575271465044]]]class
org.apache.ignite.IgniteException: GridWorker [name=ttl-cleanup-worker,
igniteInstanceName=DataStoreIgniteCache, finished=false,
heartbeatTs=1575271465044] at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
at
org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
at
org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
was:
We have Apache Ignite 3 node cluster setup with Amazon S3 Based Discovery.we
are into AWS cloud environmnet with Microservice model with 8 microservices.we
are using Ignte for session data store.While performaing load test we are
facing data grid issues if the number of clients reached 40 , Once data grid
got currepted we will lost the session store data and Application will not
respond due to session data also currepted and all the instances will auto
scaled down to intial size 8.We need to restart Apache Ignite to make the
Application up.Please find the atatched Apache Ignite Configuration for your
reference.
Apache Ignite Version : 2.7.0
> Cluster Data Store grid gets Corrupted for Load test
> ----------------------------------------------------
>
> Key: IGNITE-12456
> URL: https://issues.apache.org/jira/browse/IGNITE-12456
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.7
> Reporter: Ravi Kumar Powli
> Priority: Blocker
> Fix For: 2.8
>
> Attachments: default-config.xml
>
>
> We have Apache Ignite 3 node cluster setup with Amazon S3 Based Discovery.we
> are into AWS cloud environment with Microservice model with 8
> microservices.we are using Ignite for session data store.While performing
> load test we are facing data grid issues if the number of clients reached 40
> , Once data grid got corrupted we will lost the session store data and
> Application will not respond due to session data also corrupted and all the
> instances will auto scaled down to initial size 8.We need to restart Apache
> Ignite to make the Application up.Please find the attached Apache Ignite
> Configuration for your reference.
> Apache Ignite Version : 2.7.0
> 07:24:46,678][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-5,
> blockedFor=21s]07:24:46,678][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-5,
> blockedFor=21s][07:24:46,680][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271465499]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271465499] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:52,692][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=ttl-cleanup-worker,
> blockedFor=27s][07:24:52,692][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=ttl-cleanup-worker, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271465044]]]class
> org.apache.ignite.IgniteException: GridWorker [name=ttl-cleanup-worker,
> igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271465044] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:55,694][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-1,
> blockedFor=24s][07:24:55,694][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-1, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271471538]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-1, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271471538] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:58,700][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-4,
> blockedFor=29s][07:24:58,700][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-4, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271469397]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-4, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271469397] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:24:59,833][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-7,
> blockedFor=29s][07:24:59,833][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-7, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271469973]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-7, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271469973] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:04,716][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-0,
> blockedFor=39s][07:25:04,716][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-0, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271464782]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-0, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271464782] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:05,870][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-3,
> blockedFor=34s][07:25:05,870][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-3, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271471528]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-3, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271471528] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:07,011][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-6,
> blockedFor=41s][07:25:07,011][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-6, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271465428]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-6, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271465428] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:08,167][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-5,
> blockedFor=42s][07:25:08,167][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271465499]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-5, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271465499] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:09,310][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=sys-stripe-2,
> blockedFor=40s][07:25:09,311][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=sys-stripe-2, igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271468397]]]class org.apache.ignite.IgniteException:
> GridWorker [name=sys-stripe-2, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271468397] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)[07:25:14,183][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][G]
> Blocked system-critical thread has been detected. This can lead to
> cluster-wide undefined behaviour [threadName=ttl-cleanup-worker,
> blockedFor=49s][07:25:14,183][SEVERE][tcp-disco-msg-worker-#2%DataStoreIgniteCache%][]
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler
> [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext
> [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker
> [name=ttl-cleanup-worker, igniteInstanceName=DataStoreIgniteCache,
> finished=false, heartbeatTs=1575271465044]]]class
> org.apache.ignite.IgniteException: GridWorker [name=ttl-cleanup-worker,
> igniteInstanceName=DataStoreIgniteCache, finished=false,
> heartbeatTs=1575271465044] at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1831)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1826)
> at
> org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233)
> at
> org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2663)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7181)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119)
> at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
