I have two Ignite nodes running in my cluster. Node1 is a discovery node, and
Node2 is a regular node. During testing, we started a process on both nodes,
then took down Node1 while the process was running (kill -9). The logs on
Node2 immediately started growing at an immense rate, over 1GB of text per
minute. Upon closer examination, they are all repeated errors that are the
same.
2015-08-21 15:50:10.615 ERROR 5849 --- [125%production%]
c.l.p.i.s.impl.BaseWorkerServiceImpl : ;message=error occurred in
service;
java.lang.IllegalStateException: Queue has been removed from cache:
GridCacheQueueAdapter [cap=2147483647, collocated=false, rmvd=true]
at
org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter.onRemoved(GridCacheQueueAdapter.java:452)
at
org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter.checkRemoved(GridCacheQueueAdapter.java:428)
at
org.apache.ignite.internal.processors.datastructures.GridAtomicCacheQueueImpl.poll(GridAtomicCacheQueueImpl.java:93)
at
org.apache.ignite.internal.processors.datastructures.GridCacheQueueAdapter.poll(GridCacheQueueAdapter.java:305)
at
org.apache.ignite.internal.processors.datastructures.GridCacheQueueProxy.poll(GridCacheQueueProxy.java:655)
at
com.leonardo.platform.ignite.shared.impl.BaseWorkerServiceImpl.execute(BaseWorkerServiceImpl.java:54)
at
org.apache.ignite.internal.processors.service.GridServiceProcessor$1.run(GridServiceProcessor.java:816)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Seem's like ignite is trying to access a queue in the cluster cache, but the
cluster cache should be available even if one node goes down right?
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Lot-s-of-repeated-error-logs-when-a-node-is-killed-in-cluster-1GB-minute-tp1143.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
