Alper, thanks for clarification, this will definitely help after we get the
info I requested. This is the only way to go with the investigation.
--Yakov
2016-10-25 11:20 GMT+03:00 Alper Tekinalp <al...@evam.com>:
> Hi Yakov.
>
> I should also mention that we load cache data from one server and wait the
> data to be replicated to others. Can that cause such a situation, too?
>
> On Tue, Oct 25, 2016 at 11:14 AM, Yakov Zhdanov <yzhda...@apache.org>
> wrote:
>
>> Alper,
>>
>> There can be multiple reasons.
>>
>> Can you please reproduce the issue one more time, collect and share the
>> following with us:
>>
>> 1. collect all the logs from all the nodes - clients and servers
>> 2. take threaddumps of all JVMs (from all nodes) with jstack -l <pid>
>>
>> --Yakov
>>
>> 2016-10-25 10:49 GMT+03:00 Alper Tekinalp <al...@evam.com>:
>>
>>> Hi.
>>>
>>> There is also a few logs as :
>>>
>>> Failed to register marshalled class for more than 10 times in a row
>>> (may affect performance).
>>>
>>> Can it be releated?
>>>
>>> On Tue, Oct 25, 2016 at 10:32 AM, Alper Tekinalp <al...@evam.com> wrote:
>>>
>>>> Hi all.
>>>>
>>>> We have 3 servers and cache configuration like:
>>>>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration"
>>>> name="DEFAULT">
>>>> <property name="atomicityMode" value="ATOMIC" />
>>>> <property name="cacheMode" value="PARTITIONED" />
>>>> <property name="memoryMode" value="OFFHEAP_TIERED" />
>>>> <property name="backups" value="1" />
>>>> <property name="affinity">
>>>> <bean class="org.apache.ignite.cache
>>>> .affinity.fair.FairAffinityFunction">
>>>> <constructor-arg index="0" type="int"
>>>> value="#{evamProperties['topology.cache.partition.size']}"/>
>>>> </bean>
>>>> </property>
>>>> <!-- max memory is set to 0 means unlimited -->
>>>> <property name="offHeapMaxMemory" value="0" />
>>>> <property name="writeThrough" value="false" />
>>>> <property name="writeBehindEnabled" value="false" />
>>>> <property name="eagerTtl" value="false"/>
>>>> </bean>
>>>>
>>>> For our worker threads we check heartbeat and if a thread did not sent
>>>> heart beat for 10 minutes we consider it as stucked and interrrupt and
>>>> recreate it.
>>>>
>>>> As I can see all our worker threads are stucked in cache.put() state
>>>> and interrupted and recreated regularly.
>>>>
>>>> What can be the reason we are stucked at put? Following is stacktrace
>>>> for interruption error.
>>>>
>>>> javax.cache.CacheException: class
>>>> org.apache.ignite.IgniteInterruptedException:
>>>> Failed to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.processors.cache.GridCacheUtils.c
>>>> onvertToCacheException(GridCacheUtils.java:1502)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .cacheException(IgniteCacheProxy.java:2021)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1221)
>>>> at com.intellica.project.helper.ee.ConfigManagerHelperEE.setSta
>>>> te(ConfigManagerHelperEE.java:90)
>>>> at com.intellica.project.helper.ee.StateMachineConfigManagerEEI
>>>> mpl.store(StateMachineConfigManagerEEImpl.java:53)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.storeCon
>>>> fig(AbstractEventProcessor.java:175)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:130)
>>>> at com.evelopers.unimod.runtime.AbstractEventProcessor.process(
>>>> AbstractEventProcessor.java:80)
>>>> at com.evelopers.unimod.runtime.ModelEngine.process(ModelEngine
>>>> .java:199)
>>>> at com.evelopers.unimod.runtime.StrictHandler.handle(StrictHand
>>>> ler.java:46)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenarioLogic(AbstractScenarioWorker.java:172)
>>>> at com.intellica.evam.engine.server.worker.AbstractScenarioWork
>>>> er.runScenario(AbstractScenarioWorker.java:130)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.processE
>>>> vent(AsyncWorker.java:156)
>>>> at com.intellica.evam.engine.server.worker.AsyncWorker.run(Asyn
>>>> cWorker.java:88)
>>>> Caused by: class org.apache.ignite.IgniteInterruptedException: Failed
>>>> to wait for asynchronous operation permit (thread got interrupted).
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:747)
>>>> at org.apache.ignite.internal.util.IgniteUtils$2.apply(IgniteUt
>>>> ils.java:745)
>>>> ... 14 more
>>>> Caused by: java.lang.InterruptedException
>>>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir
>>>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
>>>> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .asyncOpAcquire(GridCacheAdapter.java:4597)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:683)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.updateAsync0(GridDhtAtomicCache.java:1014)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.putAsync0(GridDhtAtomicCache.java:484)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .putAsync(GridCacheAdapter.java:2541)
>>>> at org.apache.ignite.internal.processors.cache.distributed.dht.
>>>> atomic.GridDhtAtomicCache.put(GridDhtAtomicCache.java:461)
>>>> at org.apache.ignite.internal.processors.cache.GridCacheAdapter
>>>> .put(GridCacheAdapter.java:2215)
>>>> at org.apache.ignite.internal.processors.cache.IgniteCacheProxy
>>>> .put(IgniteCacheProxy.java:1214)
>>>> ... 11 more
>>>>
>>>>
>>>> --
>>>> Alper Tekinalp
>>>>
>>>> Software Developer
>>>> Evam Streaming Analytics
>>>>
>>>> Atatürk Mah. Turgut Özal Bulv.
>>>> Gardenya 5 Plaza K:6 Ataşehir
>>>> 34758 İSTANBUL
>>>>
>>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>>> www.evam.com.tr
>>>> <http://www.evam.com>
>>>>
>>>
>>>
>>>
>>> --
>>> Alper Tekinalp
>>>
>>> Software Developer
>>> Evam Streaming Analytics
>>>
>>> Atatürk Mah. Turgut Özal Bulv.
>>> Gardenya 5 Plaza K:6 Ataşehir
>>> 34758 İSTANBUL
>>>
>>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
>>> www.evam.com.tr
>>> <http://www.evam.com>
>>>
>>
>>
>
>
> --
> Alper Tekinalp
>
> Software Developer
> Evam Streaming Analytics
>
> Atatürk Mah. Turgut Özal Bulv.
> Gardenya 5 Plaza K:6 Ataşehir
> 34758 İSTANBUL
>
> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54
> www.evam.com.tr
> <http://www.evam.com>
>
