Alper, There can be multiple reasons.
Can you please reproduce the issue one more time, collect and share the following with us: 1. collect all the logs from all the nodes - clients and servers 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid> --Yakov 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[email protected]>: > Hi. > > There is also a few logs as : > > Failed to register marshalled class for more than 10 times in a row (may > affect performance). > > Can it be releated? > > On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[email protected]> wrote: > >> Hi all. >> >> We have 3 servers and cache configuration like: >> >> <bean class="org.apache.ignite.configuration.CacheConfiguration" >> name="DEFAULT"> >> <property name="atomicityMode" value="ATOMIC" /> >> <property name="cacheMode" value="PARTITIONED" /> >> <property name="memoryMode" value="OFFHEAP_TIERED" /> >> <property name="backups" value="1" /> >> <property name="affinity"> >> <bean class="org.apache.ignite.cache >> .affinity.fair.FairAffinityFunction"> >> <constructor-arg index="0" type="int" >> value="#{evamProperties['topology.cache.partition.size']}"/> >> </bean> >> </property> >> <!-- max memory is set to 0 means unlimited --> >> <property name="offHeapMaxMemory" value="0" /> >> <property name="writeThrough" value="false" /> >> <property name="writeBehindEnabled" value="false" /> >> <property name="eagerTtl" value="false"/> >> </bean> >> >> For our worker threads we check heartbeat and if a thread did not sent >> heart beat for 10 minutes we consider it as stucked and interrrupt and >> recreate it. >> >> As I can see all our worker threads are stucked in cache.put() state and >> interrupted and recreated regularly. >> >> What can be the reason we are stucked at put? Following is stacktrace for >> interruption error. >> >> javax.cache.CacheException: class >> org.apache.ignite.IgniteInterruptedException: >> Failed to wait for asynchronous operation permit (thread got interrupted). >> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c >> onvertToCacheException(GridCacheUtils.java:1502) >> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >> .cacheException(IgniteCacheProxy.java:2021) >> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >> .put(IgniteCacheProxy.java:1221) >> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta >> te(ConfigManagerHelperEE.java:90) >> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI >> mpl.store(StateMachineConfigManagerEEImpl.java:53) >> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon >> fig(AbstractEventProcessor.java:175) >> at com.evelopers.unimod.runtime.AbstractEventProcessor.process( >> AbstractEventProcessor.java:130) >> at com.evelopers.unimod.runtime.AbstractEventProcessor.process( >> AbstractEventProcessor.java:80) >> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine >> .java:199) >> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand >> ler.java:46) >> at com.intellica.evam.engine.server.worker.AbstractScenarioWork >> er.runScenarioLogic(AbstractScenarioWorker.java:172) >> at com.intellica.evam.engine.server.worker.AbstractScenarioWork >> er.runScenario(AbstractScenarioWorker.java:130) >> at com.intellica.evam.engine.server.worker.AsyncWorker.processE >> vent(AsyncWorker.java:156) >> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn >> cWorker.java:88) >> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to >> wait for asynchronous operation permit (thread got interrupted). >> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt >> ils.java:747) >> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt >> ils.java:745) >> ... 14 more >> Caused by: java.lang.InterruptedException >> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir >> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) >> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) >> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >> .asyncOpAcquire(GridCacheAdapter.java:4597) >> at org.apache.ignite.internal.processors.cache.distributed.dht. >> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683) >> at org.apache.ignite.internal.processors.cache.distributed.dht. >> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014) >> at org.apache.ignite.internal.processors.cache.distributed.dht. >> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484) >> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >> .putAsync(GridCacheAdapter.java:2541) >> at org.apache.ignite.internal.processors.cache.distributed.dht. >> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461) >> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >> .put(GridCacheAdapter.java:2215) >> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >> .put(IgniteCacheProxy.java:1214) >> ... 11 more >> >> >> -- >> Alper Tekinalp >> >> Software Developer >> Evam Streaming Analytics >> >> Atatürk Mah. Turgut Özal Bulv. >> Gardenya 5 Plaza K:6 Ataşehir >> 34758 İSTANBUL >> >> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 >> www.evam.com.tr >> <http://www.evam.com> >> > > > > -- > Alper Tekinalp > > Software Developer > Evam Streaming Analytics > > Atatürk Mah. Turgut Özal Bulv. > Gardenya 5 Plaza K:6 Ataşehir > 34758 İSTANBUL > > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > www.evam.com.tr > <http://www.evam.com> >
