Hi. After restarting servers, that issue did not occur again. I guess for some reason the cluster did not start normally.
Thanks responding so quick. Regards. On Tue, Oct 25, 2016 at 11:31 AM, Yakov Zhdanov <[email protected]> wrote: > Alper, thanks for clarification, this will definitely help after we get the > info I requested. This is the only way to go with the investigation. > > --Yakov > > 2016-10-25 11:20 GMT+03:00 Alper Tekinalp <[email protected]>: > > > Hi Yakov. > > > > I should also mention that we load cache data from one server and wait > the > > data to be replicated to others. Can that cause such a situation, too? > > > > On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <[email protected]> > > wrote: > > > >> Alper, > >> > >> There can be multiple reasons. > >> > >> Can you please reproduce the issue one more time, collect and share the > >> following with us: > >> > >> 1. collect all the logs from all the nodes - clients and servers > >> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid> > >> > >> --Yakov > >> > >> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <[email protected]>: > >> > >>> Hi. > >>> > >>> There is also a few logs as : > >>> > >>> Failed to register marshalled class for more than 10 times in a row > >>> (may affect performance). > >>> > >>> Can it be releated? > >>> > >>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <[email protected]> > wrote: > >>> > >>>> Hi all. > >>>> > >>>> We have 3 servers and cache configuration like: > >>>> > >>>> <bean class="org.apache.ignite.configuration.CacheConfiguration" > >>>> name="DEFAULT"> > >>>> <property name="atomicityMode" value="ATOMIC" /> > >>>> <property name="cacheMode" value="PARTITIONED" /> > >>>> <property name="memoryMode" value="OFFHEAP_TIERED" /> > >>>> <property name="backups" value="1" /> > >>>> <property name="affinity"> > >>>> <bean class="org.apache.ignite.cache > >>>> .affinity.fair.FairAffinityFunction"> > >>>> <constructor-arg index="0" type="int" > >>>> value="#{evamProperties['topology.cache.partition.size']}"/> > >>>> </bean> > >>>> </property> > >>>> <!-- max memory is set to 0 means unlimited --> > >>>> <property name="offHeapMaxMemory" value="0" /> > >>>> <property name="writeThrough" value="false" /> > >>>> <property name="writeBehindEnabled" value="false" /> > >>>> <property name="eagerTtl" value="false"/> > >>>> </bean> > >>>> > >>>> For our worker threads we check heartbeat and if a thread did not sent > >>>> heart beat for 10 minutes we consider it as stucked and interrrupt and > >>>> recreate it. > >>>> > >>>> As I can see all our worker threads are stucked in cache.put() state > >>>> and interrupted and recreated regularly. > >>>> > >>>> What can be the reason we are stucked at put? Following is stacktrace > >>>> for interruption error. > >>>> > >>>> javax.cache.CacheException: class org.apache.ignite. > IgniteInterruptedException: > >>>> Failed to wait for asynchronous operation permit (thread got > interrupted). > >>>> at org.apache.ignite.internal.processors.cache. > GridCacheUtils.c > >>>> onvertToCacheException(GridCacheUtils.java:1502) > >>>> at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy > >>>> .cacheException(IgniteCacheProxy.java:2021) > >>>> at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy > >>>> .put(IgniteCacheProxy.java:1221) > >>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE. > setSta > >>>> te(ConfigManagerHelperEE.java:90) > >>>> at com.intellica.project.helper.ee. > StateMachineConfigManagerEEI > >>>> mpl.store(StateMachineConfigManagerEEImpl.java:53) > >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor. > storeCon > >>>> fig(AbstractEventProcessor.java:175) > >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor. > process( > >>>> AbstractEventProcessor.java:130) > >>>> at com.evelopers.unimod.runtime.AbstractEventProcessor. > process( > >>>> AbstractEventProcessor.java:80) > >>>> at com.evelopers.unimod.runtime.ModelEngine.process( > ModelEngine > >>>> .java:199) > >>>> at com.evelopers.unimod.runtime.StrictHandler.handle( > StrictHand > >>>> ler.java:46) > >>>> at com.intellica.evam.engine.server.worker. > AbstractScenarioWork > >>>> er.runScenarioLogic(AbstractScenarioWorker.java:172) > >>>> at com.intellica.evam.engine.server.worker. > AbstractScenarioWork > >>>> er.runScenario(AbstractScenarioWorker.java:130) > >>>> at com.intellica.evam.engine.server.worker.AsyncWorker. > processE > >>>> vent(AsyncWorker.java:156) > >>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run( > Asyn > >>>> cWorker.java:88) > >>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed > >>>> to wait for asynchronous operation permit (thread got interrupted). > >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply( > IgniteUt > >>>> ils.java:747) > >>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply( > IgniteUt > >>>> ils.java:745) > >>>> ... 14 more > >>>> Caused by: java.lang.InterruptedException > >>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer. > acquir > >>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301) > >>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317) > >>>> at org.apache.ignite.internal.processors.cache. > GridCacheAdapter > >>>> .asyncOpAcquire(GridCacheAdapter.java:4597) > >>>> at org.apache.ignite.internal.processors.cache.distributed. > dht. > >>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683) > >>>> at org.apache.ignite.internal.processors.cache.distributed. > dht. > >>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014) > >>>> at org.apache.ignite.internal.processors.cache.distributed. > dht. > >>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484) > >>>> at org.apache.ignite.internal.processors.cache. > GridCacheAdapter > >>>> .putAsync(GridCacheAdapter.java:2541) > >>>> at org.apache.ignite.internal.processors.cache.distributed. > dht. > >>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461) > >>>> at org.apache.ignite.internal.processors.cache. > GridCacheAdapter > >>>> .put(GridCacheAdapter.java:2215) > >>>> at org.apache.ignite.internal.processors.cache. > IgniteCacheProxy > >>>> .put(IgniteCacheProxy.java:1214) > >>>> ... 11 more > >>>> > >>>> > >>>> -- > >>>> Alper Tekinalp > >>>> > >>>> Software Developer > >>>> Evam Streaming Analytics > >>>> > >>>> Atatürk Mah. Turgut Özal Bulv. > >>>> Gardenya 5 Plaza K:6 Ataşehir > >>>> 34758 İSTANBUL > >>>> > >>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > >>>> www.evam.com.tr > >>>> <http://www.evam.com> > >>>> > >>> > >>> > >>> > >>> -- > >>> Alper Tekinalp > >>> > >>> Software Developer > >>> Evam Streaming Analytics > >>> > >>> Atatürk Mah. Turgut Özal Bulv. > >>> Gardenya 5 Plaza K:6 Ataşehir > >>> 34758 İSTANBUL > >>> > >>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > >>> www.evam.com.tr > >>> <http://www.evam.com> > >>> > >> > >> > > > > > > -- > > Alper Tekinalp > > > > Software Developer > > Evam Streaming Analytics > > > > Atatürk Mah. Turgut Özal Bulv. > > Gardenya 5 Plaza K:6 Ataşehir > > 34758 İSTANBUL > > > > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > > www.evam.com.tr > > <http://www.evam.com> > > > -- Alper Tekinalp Software Developer Evam Streaming Analytics Atatürk Mah. Turgut Özal Bulv. Gardenya 5 Plaza K:6 Ataşehir 34758 İSTANBUL Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 www.evam.com.tr <http://www.evam.com>
