Hello! Do you happen to use SQL MVCC? https://apacheignite.readme.io/docs/multiversion-concurrency-control#section-multiversion-concurrency-control
Looks like you've stumbled on a previously unknown issue. Can you please file a ticket against Apache Ignite JIRA, with some description of steps to reproduce? Regards, -- Ilya Kasnacheev чт, 23 мая 2019 г. в 12:19, garima.j <[email protected]>: > Hello, > > We have a 3 node cluster in production for Apache Ignite version 2.7. RAM > 128GB. A Spark streaming service (with thick Ignite clients) writes data > into Ignite cache in a transaction (first get and then put). > Now, 3 spark clients stopped and hence NODE_FAILED events were received. My > Ignite crashed on one node and had the below stack trace : > > [2019-05-23T13:57:04,976][WARN ][sys-stripe-5-#6][lock] Received near > enlist > request from unknown node (will ignore) [txId=GridCacheVersion > [topVer=169659586, order=1558471024158, nodeOrder=23], > node=1be3bce3-7220-45bc-9863-4f16d97ea22b] > [2019-05-23T13:57:04,977][ERROR][sys-stripe-5-#6][GridCacheIoManager] > Failed > processing message [senderId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > msg=GridNearTxEnlistRequest [threadId=5872, > futId=c3170abca61-33b3ea8d-0a3e-44cb-83e6-032a37a9eed1, clientFirst=false, > miniId=1, subjId=1be3bce3-7220-45bc-9863-4f16d97ea22b, > topVer=AffinityTopologyVersion [topVer=101, minorTopVer=0], > lockVer=GridCacheVersion [topVer=169659586, order=1558471024158, > nodeOrder=23], mvccSnapshot=MvccSnapshotResponse [futId=1221240, > crdVer=1558179485875, cntr=110485182, opCntr=1, txs=[101051367, 110485176], > cleanupVer=101051361, tracking=0], timeout=5000, txTimeout=5000, > taskNameHash=0, op=UPSERT, needRes=false]] > java.lang.NullPointerException: null > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.<init>(GridDhtTxAbstractEnlistFuture.java:237) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxEnlistFuture.<init>(GridDhtTxEnlistFuture.java:84) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2061) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227) > ~[ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > [ignite-core-2.7.0.jar:2.7.0] > at > > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505) > [ignite-core-2.7.0.jar:2.7.0] > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > [ignite-core-2.7.0.jar:2.7.0] > at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201] > [2019-05-23T13:57:05,043][ERROR][sys-stripe-5-#6][] Critical system error > detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler > [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.NullPointerException]] > java.lang.NullPointerException: null > > Please help and let me know why this failure happened. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
