Hi Yakov. I should also mention that we load cache data from one server and wait the data to be replicated to others. Can that cause such a situation, too?
On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <[email protected]> wrote: > Alper, > > There can be multiple reasons. > > Can you please reproduce the issue one more time, collect and share the > following with us: > > 1. collect all the logs from all the nodes - clients and servers > 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid> > > --Yakov > > 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[email protected]>: > >> Hi. >> >> There is also a few logs as : >> >> Failed to register marshalled class for more than 10 times in a row (may >> affect performance). >> >> Can it be releated? >> >> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[email protected]> wrote: >> >>> Hi all. >>> >>> We have 3 servers and cache configuration like: >>> >>> <bean class="org.apache.ignite.configuration.CacheConfiguration" >>> name="DEFAULT"> >>> <property name="atomicityMode" value="ATOMIC" /> >>> <property name="cacheMode" value="PARTITIONED" /> >>> <property name="memoryMode" value="OFFHEAP_TIERED" /> >>> <property name="backups" value="1" /> >>> <property name="affinity"> >>> <bean class="org.apache.ignite.cache >>> .affinity.fair.FairAffinityFunction"> >>> <constructor-arg index="0" type="int" >>> value="#{evamProperties['topology.cache.partition.size']}"/> >>> </bean> >>> </property> >>> <!-- max memory is set to 0 means unlimited --> >>> <property name="offHeapMaxMemory" value="0" /> >>> <property name="writeThrough" value="false" /> >>> <property name="writeBehindEnabled" value="false" /> >>> <property name="eagerTtl" value="false"/> >>> </bean> >>> >>> For our worker threads we check heartbeat and if a thread did not sent >>> heart beat for 10 minutes we consider it as stucked and interrrupt and >>> recreate it. >>> >>> As I can see all our worker threads are stucked in cache.put() state and >>> interrupted and recreated regularly. >>> >>> What can be the reason we are stucked at put? Following is stacktrace >>> for interruption error. >>> >>> javax.cache.CacheException: class >>> org.apache.ignite.IgniteInterruptedException: >>> Failed to wait for asynchronous operation permit (thread got interrupted). >>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c >>> onvertToCacheException(GridCacheUtils.java:1502) >>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >>> .cacheException(IgniteCacheProxy.java:2021) >>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >>> .put(IgniteCacheProxy.java:1221) >>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta >>> te(ConfigManagerHelperEE.java:90) >>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI >>> mpl.store(StateMachineConfigManagerEEImpl.java:53) >>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon >>> fig(AbstractEventProcessor.java:175) >>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process( >>> AbstractEventProcessor.java:130) >>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process( >>> AbstractEventProcessor.java:80) >>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine >>> .java:199) >>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand >>> ler.java:46) >>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork >>> er.runScenarioLogic(AbstractScenarioWorker.java:172) >>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork >>> er.runScenario(AbstractScenarioWorker.java:130) >>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE >>> vent(AsyncWorker.java:156) >>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn >>> cWorker.java:88) >>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed >>> to wait for asynchronous operation permit (thread got interrupted). >>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt >>> ils.java:747) >>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt >>> ils.java:745) >>> ... 14 more >>> Caused by: java.lang.InterruptedException >>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir >>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) >>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) >>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> .asyncOpAcquire(GridCacheAdapter.java:4597) >>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683) >>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014) >>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484) >>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> .putAsync(GridCacheAdapter.java:2541) >>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461) >>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter >>> .put(GridCacheAdapter.java:2215) >>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy >>> .put(IgniteCacheProxy.java:1214) >>> ... 11 more >>> >>> >>> -- >>> Alper Tekinalp >>> >>> Software Developer >>> Evam Streaming Analytics >>> >>> Atatürk Mah. Turgut Özal Bulv. >>> Gardenya 5 Plaza K:6 Ataşehir >>> 34758 İSTANBUL >>> >>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 >>> www.evam.com.tr >>> <http://www.evam.com> >>> >> >> >> >> -- >> Alper Tekinalp >> >> Software Developer >> Evam Streaming Analytics >> >> Atatürk Mah. Turgut Özal Bulv. >> Gardenya 5 Plaza K:6 Ataşehir >> 34758 İSTANBUL >> >> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 >> www.evam.com.tr >> <http://www.evam.com> >> > > -- Alper Tekinalp Software Developer Evam Streaming Analytics Atatürk Mah. Turgut Özal Bulv. Gardenya 5 Plaza K:6 Ataşehir 34758 İSTANBUL Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 www.evam.com.tr <http://www.evam.com>
