Hi. There is also a few logs as :
Failed to register marshalled class for more than 10 times in a row (may affect performance). Can it be releated? On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[email protected]> wrote: > Hi all. > > We have 3 servers and cache configuration like: > > <bean class="org.apache.ignite.configuration.CacheConfiguration" > name="DEFAULT"> > <property name="atomicityMode" value="ATOMIC" /> > <property name="cacheMode" value="PARTITIONED" /> > <property name="memoryMode" value="OFFHEAP_TIERED" /> > <property name="backups" value="1" /> > <property name="affinity"> > <bean class="org.apache.ignite.cache.affinity.fair. > FairAffinityFunction"> > <constructor-arg index="0" type="int" > value="#{evamProperties['topology.cache.partition.size']}"/> > </bean> > </property> > <!-- max memory is set to 0 means unlimited --> > <property name="offHeapMaxMemory" value="0" /> > <property name="writeThrough" value="false" /> > <property name="writeBehindEnabled" value="false" /> > <property name="eagerTtl" value="false"/> > </bean> > > For our worker threads we check heartbeat and if a thread did not sent > heart beat for 10 minutes we consider it as stucked and interrrupt and > recreate it. > > As I can see all our worker threads are stucked in cache.put() state and > interrupted and recreated regularly. > > What can be the reason we are stucked at put? Following is stacktrace for > interruption error. > > javax.cache.CacheException: class > org.apache.ignite.IgniteInterruptedException: > Failed to wait for asynchronous operation permit (thread got interrupted). > at org.apache.ignite.internal.processors.cache.GridCacheUtils. > convertToCacheException(GridCacheUtils.java:1502) > at org.apache.ignite.internal.processors.cache.IgniteCacheProxy. > cacheException(IgniteCacheProxy.java:2021) > at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy.put(IgniteCacheProxy.java:1221) > at com.intellica.project.helper.ee.ConfigManagerHelperEE.setState( > ConfigManagerHelperEE.java:90) > at com.intellica.project.helper.ee.StateMachineConfigManagerEEImp > l.store(StateMachineConfigManagerEEImpl.java:53) > at com.evelopers.unimod.runtime.AbstractEventProcessor. > storeConfig(AbstractEventProcessor.java:175) > at com.evelopers.unimod.runtime.AbstractEventProcessor.process( > AbstractEventProcessor.java:130) > at com.evelopers.unimod.runtime.AbstractEventProcessor.process( > AbstractEventProcessor.java:80) > at com.evelopers.unimod.runtime.ModelEngine.process( > ModelEngine.java:199) > at com.evelopers.unimod.runtime.StrictHandler.handle( > StrictHandler.java:46) > at com.intellica.evam.engine.server.worker.AbstractScenarioWorker. > runScenarioLogic(AbstractScenarioWorker.java:172) > at com.intellica.evam.engine.server.worker.AbstractScenarioWorker. > runScenario(AbstractScenarioWorker.java:130) > at com.intellica.evam.engine.server.worker.AsyncWorker. > processEvent(AsyncWorker.java:156) > at com.intellica.evam.engine.server.worker.AsyncWorker.run( > AsyncWorker.java:88) > Caused by: class org.apache.ignite.IgniteInterruptedException: Failed to > wait for asynchronous operation permit (thread got interrupted). > at org.apache.ignite.internal.util.IgniteUtils$2.apply( > IgniteUtils.java:747) > at org.apache.ignite.internal.util.IgniteUtils$2.apply( > IgniteUtils.java:745) > ... 14 more > Caused by: java.lang.InterruptedException > at java.util.concurrent.locks.AbstractQueuedSynchronizer. > acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) > at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) > at org.apache.ignite.internal.processors.cache.GridCacheAdapter. > asyncOpAcquire(GridCacheAdapter.java:4597) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484) > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.putAsync(GridCacheAdapter.java:2541) > at org.apache.ignite.internal.processors.cache.distributed. > dht.atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461) > at org.apache.ignite.internal.processors.cache. > GridCacheAdapter.put(GridCacheAdapter.java:2215) > at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy.put(IgniteCacheProxy.java:1214) > ... 11 more > > > -- > Alper Tekinalp > > Software Developer > Evam Streaming Analytics > > Atatürk Mah. Turgut Özal Bulv. > Gardenya 5 Plaza K:6 Ataşehir > 34758 İSTANBUL > > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > www.evam.com.tr > <http://www.evam.com> > -- Alper Tekinalp Software Developer Evam Streaming Analytics Atatürk Mah. Turgut Özal Bulv. Gardenya 5 Plaza K:6 Ataşehir 34758 İSTANBUL Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 www.evam.com.tr <http://www.evam.com>
