Hi all.
We have 3 servers and cache configuration like:
<bean class="org.apache.ignite.configuration.CacheConfiguration"
name="DEFAULT">
<property name="atomicityMode" value="ATOMIC" />
<property name="cacheMode" value="PARTITIONED" />
<property name="memoryMode" value="OFFHEAP_TIERED" />
<property name="backups" value="1" />
<property name="affinity">
<bean
class="org.apache.ignite.cache.affinity.fair.FairAffinityFunction">
<constructor-arg index="0" type="int"
value="#{evamProperties['topology.cache.partition.size']}"/>
</bean>
</property>
<!-- max memory is set to 0 means unlimited -->
<property name="offHeapMaxMemory" value="0" />
<property name="writeThrough" value="false" />
<property name="writeBehindEnabled" value="false" />
<property name="eagerTtl" value="false"/>
</bean>
For our worker threads we check heartbeat and if a thread did not sent
heart beat for 10 minutes we consider it as stucked and interrrupt and
recreate it.
As I can see all our worker threads are stucked in cache.put() state and
interrupted and recreated regularly.
What can be the reason we are stucked at put? Following is stacktrace for
interruption error.
javax.cache.CacheException: class
org.apache.ignite.IgniteInterruptedException: Failed to wait for
asynchronous operation permit (thread got interrupted).
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1502)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:2021)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1221)
at
com.intellica.project.helper.ee.ConfigManagerHelperEE.setState(ConfigManagerHelperEE.java:90)
at
com.intellica.project.helper.ee.StateMachineConfigManagerEEImpl.store(StateMachineConfigManagerEEImpl.java:53)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.storeConfig(AbstractEventProcessor.java:175)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:130)
at
com.evelopers.unimod.runtime.AbstractEventProcessor.process(AbstractEventProcessor.java:80)
at
com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine.java:199)
at
com.evelopers.unimod.runtime.StrictHandler.handle(StrictHandler.java:46)
at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenarioLogic(AbstractScenarioWorker.java:172)
at
com.intellica.evam.engine.server.worker.AbstractScenarioWorker.runScenario(AbstractScenarioWorker.java:130)
at
com.intellica.evam.engine.server.worker.AsyncWorker.processEvent(AsyncWorker.java:156)
at
com.intellica.evam.engine.server.worker.AsyncWorker.run(AsyncWorker.java:88)
Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to
wait for asynchronous operation permit (thread got interrupted).
at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:747)
at
org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUtils.java:745)
... 14 more
Caused by: java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOpAcquire(GridCacheAdapter.java:4597)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAsync(GridCacheAdapter.java:2541)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
at
org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2215)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.put(IgniteCacheProxy.java:1214)
... 11 more
--
Alper Tekinalp
Software Developer
Evam Streaming Analytics
Atatürk Mah. Turgut Özal Bulv.
Gardenya 5 Plaza K:6 Ataşehir
34758 İSTANBUL
Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
www.evam.com.tr
<http://www.evam.com>
