Hi Denis,
I saw the following error message in the logs on one of my ignite nodes
(ml-ignite1.mobilink.osa):
02/01/2018 10:39:12 WARN [tcp-disco-msg-worker-#2%jazz-prod-triggers-anum%]
TcpDiscoverySpi: Local node has detected failed nodes and started
cluster-wide procedure. To speed up failure detection please see 'Failure
Detection' section under javadoc for 'TcpDiscoverySpi'
02/01/2018 10:39:12 WARN [disco-event-worker-#105%jazz-prod-triggers-anum%]
GridDiscoveryManager: Node FAILED: TcpDiscoveryNode
[id=07826b34-d933-4400-ba3b-5270dc5df63d, addrs=[10.145.1.14, 127.0.0.1],
sockAddrs=[/127.0.0.1:47500, ml-ignite2.mobilink.osa/10.145.1.14:47500],
discPort=47500, order=1, intOrder=1, lastExchangeTime=1514888749012,
loc=false, ver=2.1.0#20170720-sha1:a6ca5c8a, isClient=false]
02/01/2018 10:39:12 INFO [disco-event-worker-#105%jazz-prod-triggers-anum%]
GridDiscoveryManager: Topology snapshot [ver=3, servers=1, clients=0,
CPUs=56, heap=10.0GB]
Looks like ignite2 node that was part of the cluster went offline. The error
message on the ignite2 node was:
02/01/2018 11:03:20 ERROR [sys-stripe-18-#19%jazz-prod-triggers-anum%]
GridEventStorageManager: Unexpected exception in listener notification for
event: CacheEvent [cacheName=CACHE_L2_DPI, part=289,
key=7a442a957218405ab9d04b8080a9f1e9, xid=null, lockId=GridCacheVersion
[topVer=126371000, order=1514890998457, nodeOrder=1], newVal=L2_DPI
[id=7a442a957218405ab9d04b8080a9f1e9, srcId=TR_DPI_G,
hasParsingErrors=FALSE, AFK_MSISDN=923089045308, MSISDN=923089045308,
IMSI=410018407922922, UPLINK_TRAFFIC=120, DOWNLINK_TRAFFIC=151,
STARTTIMESECOND=1514832983, PROTOCOL_CATEGORY=4, APPLICATION=250,
RAT_TYPE=1, IMEISV=357056089204360, TAC=35705608, CGI=, SAI=41001F0D3FF6C4,
ECGI=, TAI=, entryTime=1514891000, isProcessed=FALSE], oldVal=null,
hasOldVal=false, hasNewVal=true, near=false,
subjId=d148804e-1cce-4c6e-900f-8c7ee017715d, cloClsName=null, taskName=null,
nodeId8=d148804e, evtNodeId8=d148804e, msg=Cache event.,
type=CACHE_OBJECT_PUT, tstamp=1514891000479]
java.lang.ClassCastException:
com.thinkbiganalytics.veon.trigger.modules.common.models.L2.L2_DPI cannot be
cast to
com.thinkbiganalytics.veon.trigger.modules.common.models.L1.L1_TriggerRecord
at
com.thinkbiganalytics.veon.trigger.modules.core.controller.events.router.RouterLogic.process(RouterLogic.java:39)
at
com.thinkbiganalytics.veon.trigger.modules.core.controller.events.listener.ListenerLocalCache$1.apply(ListenerLocalCache.java:33)
at
com.thinkbiganalytics.veon.trigger.modules.core.controller.events.listener.ListenerLocalCache$1.apply(ListenerLocalCache.java:24)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager$UserListenerWrapper.onEvent(GridEventStorageManager.java:1438)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:895)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record0(GridEventStorageManager.java:344)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:301)
at
org.apache.ignite.internal.processors.cache.GridCacheEventManager.addEvent(GridCacheEventManager.java:327)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1794)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2437)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1847)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$2900(GridDhtAtomicCache.java:129)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$30.apply(GridDhtAtomicCache.java:1703)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$30.apply(GridDhtAtomicCache.java:1689)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onDone(GridDhtForceKeysFuture.java:153)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtForceKeysFuture.onDone(GridDhtForceKeysFuture.java:69)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:278)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:148)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
