[
https://issues.apache.org/jira/browse/IGNITE-15886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gaurav Aggarwal updated IGNITE-15886:
-------------------------------------
Description:
{+}*Reproducer*{+}:
Run a cluster with few nodes, on bringing down one of the nodes, the other
nodes throw this exception intermittently and eventually come down
+*Actual result*+
Intermittent exception in one of the nodes:
{noformat}
SEVERE: Failed to notify direct custom event listener:
StartRoutineDiscoveryMessage [startReqData=StartRequestData
[prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@3a0d7950,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
[rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
autoUnsubscribe=true], keepBinary=false, deserEx=null,
routineId=a452f5c7-4a27-4a5e-be3e-77ddae5d50a3]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)Nov 03, 2021 12:11:00 PM
org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to notify direct custom event listener:
StartRoutineDiscoveryMessage [startReqData=StartRequestData
[prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@67cc413e,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
[rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
autoUnsubscribe=true], keepBinary=false, deserEx=null,
routineId=c2ce7d01-5c5b-4654-8942-bd7ab6b90351]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
{noformat}
was:
{+}*Reproducer*{+}:
Run a cluster with few nodes, on bringing down one of the nodes, the other
nodes throw this exception intermittently and eventually come down
+*Actual result*+
Intermittent exception in one of the nodes:
{noformat}
[16:54:38,758][SEVERE][disco-notifier-worker-#43%CashFlowCluster_16b67e98563f4cfbac95ae055a00e67f%][GridDiscoveryManager]
Failed to notify direct custom event listener: StartRoutineDiscoveryMessage
[startReqData=StartRequestData
[prjPred=sbt.cashflow.grid.services.cachefactory.ignite.NodeAttributeFilter@63ae71a9,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandler
[returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@594bf5b8,
cacheName=CALC_REQUESTS, rmtFilter=null, rmtFilterDep=null, internal=false,
notifyExisting=false, oldValRequired=true, sync=false, ignoreExpired=true,
taskHash=0, skipPrimaryCheck=false, locOnly=false, keepBinary=true,
ackBuf=null, cacheId=-1608655250, initTopVer=null, nodeLeft=false,
ignoreClsNotFound=false, nodeId=null, routineId=null], bufSize=1, interval=0,
autoUnsubscribe=true], keepBinary=true,
routineId=021dd2ce-3d8a-41c1-a4d0-b625ea1284f4]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:82)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:96)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1424)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:110)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:202)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:193)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:722)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:601)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2683)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2721)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:745)
[16:54:39,725][SEVERE][disco-notifier-worker-#43%CashFlowCluster_16b67e98563f4cfbac95ae055a00e67f%][GridDiscoveryManager]
Failed to notify direct custom event listener: StartRoutineDiscoveryMessage
[startReqData=StartRequestData
[prjPred=sbt.cashflow.grid.services.cachefactory.ignite.NodeAttributeFilter@7462c96c,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandler
[returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@6451dd70,
cacheName=DISTRIBUTED_REQUESTS, rmtFilter=null, rmtFilterDep=null,
internal=false, notifyExisting=false, oldValRequired=true, sync=false,
ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false,
keepBinary=true, ackBuf=null, cacheId=1419803136, initTopVer=null,
nodeLeft=false, ignoreClsNotFound=false, nodeId=null, routineId=null],
bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=true,
routineId=1fca5f04-d220-49ac-850a-0d4527e22eef]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:82)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:96)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1424)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:110)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:202)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:193)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:722)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:601)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2683)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2721)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:745)
[16:54:40,809][SEVERE][disco-notifier-worker-#43%CashFlowCluster_16b67e98563f4cfbac95ae055a00e67f%][GridDiscoveryManager]
Failed to notify direct custom event listener: StartRoutineDiscoveryMessage
[startReqData=StartRequestData
[prjPred=sbt.cashflow.grid.services.cachefactory.ignite.NodeAttributeFilter@4a29e4c8,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandler
[returnValTrans=o.a.i.i.processors.cache.query.continuous.CacheContinuousQueryHandler$1@28627d48,
cacheName=DISTRIBUTED_REQUESTS, rmtFilter=null, rmtFilterDep=null,
internal=false, notifyExisting=false, oldValRequired=true, sync=false,
ignoreExpired=true, taskHash=0, skipPrimaryCheck=false, locOnly=false,
keepBinary=true, ackBuf=null, cacheId=1419803136, initTopVer=null,
nodeLeft=false, ignoreClsNotFound=false, nodeId=null, routineId=null],
bufSize=1, interval=0, autoUnsubscribe=true], keepBinary=true,
routineId=aa0bdf4f-bfdb-4eb3-8d99-6bcb67532704]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:82)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:96)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1424)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:110)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:202)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:193)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:722)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:601)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2683)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2721)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
at java.lang.Thread.run(Thread.java:745)
SEVERE: Failed to notify direct custom event listener:
StartRoutineDiscoveryMessage [startReqData=StartRequestData
[prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@3a0d7950,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
[rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
autoUnsubscribe=true], keepBinary=false, deserEx=null,
routineId=a452f5c7-4a27-4a5e-be3e-77ddae5d50a3]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)Nov 03, 2021 12:11:00 PM
org.apache.ignite.logger.java.JavaLogger error
SEVERE: Failed to notify direct custom event listener:
StartRoutineDiscoveryMessage [startReqData=StartRequestData
[prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@67cc413e,
clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
[rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
autoUnsubscribe=true], keepBinary=false, deserEx=null,
routineId=c2ce7d01-5c5b-4654-8942-bd7ab6b90351]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
at
org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
{noformat}
> Intermittent [Failed to notify direct custom event listener] exception on
> node shutdown
> ---------------------------------------------------------------------------------------
>
> Key: IGNITE-15886
> URL: https://issues.apache.org/jira/browse/IGNITE-15886
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.9
> Reporter: Gaurav Aggarwal
> Assignee: Mikhail Petrov
> Priority: Major
>
> {+}*Reproducer*{+}:
> Run a cluster with few nodes, on bringing down one of the nodes, the other
> nodes throw this exception intermittently and eventually come down
> +*Actual result*+
> Intermittent exception in one of the nodes:
> {noformat}
> SEVERE: Failed to notify direct custom event listener:
> StartRoutineDiscoveryMessage [startReqData=StartRequestData
> [prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@3a0d7950,
> clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
> [rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
> autoUnsubscribe=true], keepBinary=false, deserEx=null,
> routineId=a452f5c7-4a27-4a5e-be3e-77ddae5d50a3]
> java.lang.NullPointerException
> at
> org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
> at
> org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)Nov 03, 2021 12:11:00 PM
> org.apache.ignite.logger.java.JavaLogger error
> SEVERE: Failed to notify direct custom event listener:
> StartRoutineDiscoveryMessage [startReqData=StartRequestData
> [prjPred=com.bfm.libignite.cluster.filter.ClusterGroupFilter@67cc413e,
> clsName=null, depInfo=null, hnd=CacheContinuousQueryHandlerV2
> [rmtFilterFactoryDep=null, types=0], bufSize=1, interval=0,
> autoUnsubscribe=true], keepBinary=false, deserEx=null,
> routineId=c2ce7d01-5c5b-4654-8942-bd7ab6b90351]
> java.lang.NullPointerException
> at
> org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:95)
> at
> org.apache.ignite.internal.processors.continuous.StartRoutineDiscoveryMessage.addUpdateCounters(StartRoutineDiscoveryMessage.java:109)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.processStartRequest(GridContinuousProcessor.java:1482)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.access$400(GridContinuousProcessor.java:117)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:220)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$2.onCustomEvent(GridContinuousProcessor.java:211)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:717)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:526)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2677)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2715)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
