Hi,
We have observed one issue with IGNITE today is , the client thread has been
waiting forever when it is trying to create a cache, inserting the data and
trying to know the size of the cache [ REPLICATED caches, backups value for
these caches is 2147483647 ]. We are getting this issue few times only, but
daily one or 2 times it is coming in our environment. Following are the
logs, can any one please check and suggest us.
Java Version : 1.7.0_45 [ we can't upgrade to newer version.]
Ignite Version : 1.8
Ignite server nodes = 3
Unique Ignite Client services = 6
cache configuration:
<bean id="lMJPGrpsAvailableInTemplateCacheIgnite"
class="org.apache.ignite.configuration.CacheConfiguration"
scope="singleton">
<property name="name"
value="lMJPGrpsAvailableInTemplateCacheIgnite"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="memoryMode" value="ONHEAP_TIERED"/>
<property name="offHeapMaxMemory" value="#{1 * 20L * 1024L *
1024L}"/>
<property name="backups" value="2147483647"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="startSize" value="#{5 * 1024 * 1024}"/>
<property name="swapEnabled" value="false"/>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="0"/>
<property name="rebalanceThreadPoolSize" value="4"/>
</bean>
<bean id="lMAssoMappingCacheIgnite"
class="org.apache.ignite.configuration.CacheConfiguration"
scope="singleton">
<property name="name" value="lMAssoMappingCacheIgnite"/>
<property name="cacheMode" value="REPLICATED"/>
<property name="memoryMode" value="ONHEAP_TIERED"/>
<property name="offHeapMaxMemory" value="#{1 * 20L * 1024L *
1024L}"/>
<property name="backups" value="2147483647"/>
<property name="writeSynchronizationMode" value="FULL_SYNC"/>
<property name="startSize" value="#{5 * 1024 * 1024}"/>
<property name="swapEnabled" value="false"/>
<property name="rebalanceMode" value="ASYNC"/>
<property name="rebalanceBatchSize" value="#{1024 * 1024}"/>
<property name="rebalanceThrottle" value="0"/>
<property name="rebalanceThreadPoolSize" value="4"/>
</bean>
DS - Client Service 1:
Thread stack trace from thread dump When first time it is occurred:
-----
when we are trying to sync the data from DS service to ignite server, thread
has been waiting from client is forever. Following is the stack trace from
the thread dump of this client service.
"pool-160-thread-1" prio=10 tid=0x00007f39800bc800 nid=0x404e waiting on
condition [0x00007f396f9f8000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000078ac134c8> (a
org.apache.ignite.internal.ComputeTaskInternalFuture)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:160)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:118)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.size(GridCacheAdapter.java:3833)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.size(IgniteCacheProxy.java:997)
at
com.pg.casino.service.datasync.DataSyncManager.clearFeedCaches(DataSyncManager.java:446)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:119)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:21)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Thread stack trace from thread dump When 2nd time it is occurred:
-----
When it is occurred first time, we have restarted this DS client service and
again it has started to do the same task, but at that time it took 20 mins
of time to insert the 30 rows of data into ignite. Following are the logs of
DS service. We spawn 2 threads to from this client service and insert the
data into different caches in ignite. There is no other service will insert
or modify the data in ignite server other than this DS service.
pool-4-thread-2:
09:38:08,632 - INFO - pool-4-thread-2 - IG_LOGGER DSIgniteCacheManager- got
cache mapping for-bt
09:38:07,283 - INFO - pool-4-thread-2 - DS_LOGGER DataSyncManager->syncBL()
is started
thread stack trace in thread dump:
"pool-4-thread-2" prio=10 tid=0x0000000000651800 nid=0x6619 waiting on
condition [0x00007fd25e232000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000790c4df90> (a
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:160)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:118)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.tryGetAndPut(GridDhtAtomicCache.java:632)
at
org.apache.ignite.internal.MarshallerContextImpl.registerClassName(MarshallerContextImpl.java:160)
at
org.apache.ignite.internal.MarshallerContextAdapter.registerClass(MarshallerContextAdapter.java:157)
at
org.apache.ignite.internal.binary.BinaryContext.registerUserClassDescriptor(BinaryContext.java:767)
at
org.apache.ignite.internal.binary.BinaryContext.registerClassDescriptor(BinaryContext.java:744)
at
org.apache.ignite.internal.binary.BinaryContext.descriptorForClass(BinaryContext.java:616)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal0(BinaryWriterExImpl.java:164)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:147)
at
org.apache.ignite.internal.binary.BinaryWriterExImpl.marshal(BinaryWriterExImpl.java:134)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.marshal(GridBinaryMarshaller.java:239)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.marshalToBinary(CacheObjectBinaryProcessorImpl.java:522)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toBinary(CacheObjectBinaryProcessorImpl.java:915)
at
org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:860)
at
org.apache.ignite.internal.processors.cache.GridCacheContext.toCacheObject(GridCacheContext.java:1810)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapUpdate(GridNearAtomicUpdateFuture.java:830)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:654)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:547)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:208)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$22.apply(GridDhtAtomicCache.java:1109)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$22.apply(GridDhtAtomicCache.java:1107)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:846)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsync0(GridDhtAtomicCache.java:1107)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAllAsync(GridDhtAtomicCache.java:709)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAll(GridDhtAtomicCache.java:704)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.putAll(IgniteCacheProxy.java:1423)
at
com.pg.casino.service.datasync.DataSyncManager.syncDataIntoIgnite(DataSyncManager.java:729)
at
com.pg.casino.service.datasync.DataSyncManager.syncData(DataSyncManager.java:794)
at
com.pg.casino.service.datasync.DataSyncManager.syncBLs(DataSyncManager.java:1077)
at
com.pg.casino.service.datasync.DataSyncManager.syncBLs(DataSyncManager.java:978)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:110)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:21)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
pool-4-thread-1:
09:42:17,875 - INFO - pool-4-thread-1 - LS_LOGGER
DSIgniteCacheManager::lMAssoMappingDSCache size 0
>>>>> thread is hung here for 22 mins after that it has come back<<<<<<
10:04:54,179 - INFO - pool-4-thread-1 - DS_LOGGER
DataSyncManager.clearFeedCaches-sizeBeforeClear-137
10:04:54,180 - INFO - pool-4-thread-1 - DS_LOGGER Removing entries from
Filtered feed cache
10:04:54,183 - INFO - pool-4-thread-1 - LS_LOGGER
CLSIgniteCacheManager.getKeysFromFilterFeed
Thread stack trace in thread dump:
"pool-4-thread-1" prio=10 tid=0x0000000000616000 nid=0x6618 waiting on
condition [0x00007fd25d770000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000790dfdb40> (a
org.apache.ignite.internal.processors.cache.GridCacheProcessor$DynamicCacheStartFuture)
at
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:160)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:118)
at
org.apache.ignite.internal.IgniteKernal.getOrCreateCache(IgniteKernal.java:2586)
at
com.casino.ignite.common.cache.IgniteManager.getOrCreateIgniteCache(IgniteManager.java:46)
at
com.pg.casino.service.datasync.DataSyncManager.clearFeedCaches(DataSyncManager.java:445)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:103)
at
com.pg.casino.service.datasync.DataSyncCallable.call(DataSyncCallable.java:21)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Ignite Server Logs
---------------------
At this point of time we have observed the below exception in ignite server
logs.
[09:38:13,706][WARN
][marshaller-cache-#4962%CasinoApacheIgniteServices%][TcpCommunicationSpi]
Connect timed out (consider increasing 'connTimeout' configura
tion property) [addr=gi2p1xrcnw003.gi02.bpty/10.166.130.32:9090,
connTimeout=5000]
[09:38:13,706][WARN
][marshaller-cache-#4962%CasinoApacheIgniteServices%][TcpCommunicationSpi]
Connect timed out (consider increasing 'connTimeout' configura
tion property) [addr=gi2p1xrcnw003.gi02.bpty/10.166.130.32:9090,
connTimeout=5000]
[09:38:25,831][ERROR][marshaller-cache-#4962%CasinoApacheIgniteServices%][CacheContinuousQueryHandler]
Failed to send event notification to node: 61623f48-04
68-4ba3-aa32-5d9dbcf62181
org.apache.ignite.IgniteCheckedException: Failed to send message (node may
have left the grid or TCP connection cannot be established due to firewall
issues) [node=TcpDiscoveryNode [id=61623f48-0468-4ba3-aa32-5d9dbcf62181,
addrs=[10.166.130.32, 127.0.0.1],
sockAddrs=[gi2p1xrcnw003.gi02.bpty/180.16.10.32:0, /127.0.0.1:0],
discPort=0, order=138, intOrder=83, lastExchangeTime=1523349859761,
loc=false, ver=1.8.0#20161205-sha1:9ca40dbe, isClient=true], topic=T4
[topic=TOPIC_CACHE, id1=1fd3a002-42a8-3e13-a1aa-bf164b7f2d64,
id2=61623f48-0468-4ba3-aa32-5d9dbcf62181, id3=1], msg=GridContinuousMessage
[type=MSG_EVT_NOTIFICATION, routineId=f6e3128b-899a-47e3-a69e-df5dc33910a5,
data=null, futId=null], policy=2]
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1153)
~[ignite-core-1.8.0.jar:1.8.0]
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessage(GridIoManager.java:1384)
~[ignite-core-1.8.0.jar:1.8.0]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1335)
~[ignite-core-1.8.0.jar:1.8.0]
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor.sendWithRetries(GridContinuousProcessor.java:1306)
~[ignite-core-1.8.0.ja
Thanks,
Praveen
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
