Hi Alper, Seems, sometime stripe-pool threads stucks for awhile due to short network issues. Possibly, it happens when nodes doesn't left the grid gracefully.
Try to set java.net.preferIPv4Stack=true on all nodes (incl. clients), it can help in some cases when client is droped from grid due to IPv6 connectivity issues. On Mon, Jul 17, 2017 at 11:04 AM, Alper Tekinalp <[email protected]> wrote: > Hi Andrey. > > Logs for two server nodes are attached. > > On Fri, Jul 14, 2017 at 8:41 PM, Andrey Mashenkov < > [email protected]> wrote: > >> Hi Alper, >> >> Stripe pool threads do message processing. >> In stacktrace, I see node can send message to other node due to network >> issue or node failure. >> Thread try to reconnect to the node during failure detection timeout. >> >> What Ignite version do you use? >> Would you please share full logs? >> >> On Fri, Jul 14, 2017 at 1:24 PM, Alper Tekinalp <[email protected]> wrote: >> >>> Hi. >>> >>> What does following log means: >>> >>> [WARN ] 2017-07-12 23:00:50.786 [grid-timeout-worker-#71%cache-server%] >>> G - >>> Possible starvation in striped pool: sys-stripe-10-#11%cache-server >>> % >>> [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, >>> topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, >>> msg=GridNearSingleGetRequest >>> .... >>> 43f6-946a-6593f8dcde7d, taskNameHash=0, createTtl=0, accessTtl=-1]]]] >>> deadlock: false >>> completed: 699284 >>> Thread [name="sys-stripe-10-#11%cache-server%", id=44, state=WAITING, >>> blockCnt=18197, waitCnt=642544] >>> Lock [object=o.a.i.spi.communication.tcp.TcpCommunicationSpi$Conn >>> ectFuture@604f22f, ownerName=null, ownerId=-1] >>> at sun.misc.Unsafe.park(Native Method) >>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java >>> :186) >>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAn >>> dCheckInterrupt(AbstractQueuedSynchronizer.java:834) >>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcqu >>> ireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) >>> at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquir >>> eSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) >>> at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter >>> .java:161) >>> at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter. >>> java:119) >>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.reserveClien >>> t(TcpCommunicationSpi.java:2515) >>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage0 >>> (TcpCommunicationSpi.java:2340) >>> at o.a.i.spi.communication.tcp.TcpCommunicationSpi.sendMessage( >>> TcpCommunicationSpi.java:2304) >>> at o.a.i.i.managers.communication.GridIoManager.send(GridIoMana >>> ger.java:1281) >>> at o.a.i.i.managers.communication.GridIoManager.sendOrderedMess >>> age(GridIoManager.java:1517) >>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi >>> thRetries(GridContinuousProcessor.java:1335) >>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi >>> thRetries(GridContinuousProcessor.java:1306) >>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendWi >>> thRetries(GridContinuousProcessor.java:1288) >>> at o.a.i.i.processors.continuous.GridContinuousProcessor.sendNo >>> tification(GridContinuousProcessor.java:949) >>> at o.a.i.i.processors.continuous.GridContinuousProcessor.addNot >>> ification(GridContinuousProcessor.java:892) >>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue >>> ryHandler.onEntryUpdate(CacheContinuousQueryHandler.java:814) >>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue >>> ryHandler.access$800(CacheContinuousQueryHandler.java:91) >>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue >>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:426) >>> at o.a.i.i.processors.cache.query.continuous.CacheContinuousQue >>> ryHandler$1$1.apply(CacheContinuousQueryHandler.java:421) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:450) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cAbstractUpdateFuture.onDone(GridDhtAtomicAbstractUpdateFuture.java:56) >>> at o.a.i.i.util.future.GridFutureAdapter.onDone(GridFutureAdapt >>> er.java:321) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cAbstractUpdateFuture.registerResponse(GridDhtAtomicAbstract >>> UpdateFuture.java:343) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cAbstractUpdateFuture.onResult(GridDhtAtomicAbstractUpdateFu >>> ture.java:399) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cCache.processDhtAtomicDeferredUpdateResponse(GridDhtAtomicC >>> ache.java:3388) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cCache.access$2000(GridDhtAtomicCache.java:126) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cCache$10.apply(GridDhtAtomicCache.java:412) >>> at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomi >>> cCache$10.apply(GridDhtAtomicCache.java:407) >>> at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(G >>> ridCacheIoManager.java:827) >>> at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridC >>> acheIoManager.java:369) >>> at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(Gr >>> idCacheIoManager.java:293) >>> at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridC >>> acheIoManager.java:95) >>> at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(Grid >>> CacheIoManager.java:238) >>> at o.a.i.i.managers.communication.GridIoManager.invokeListener( >>> GridIoManager.java:1215) >>> at o.a.i.i.managers.communication.GridIoManager.processRegularM >>> essage0(GridIoManager.java:843) >>> at o.a.i.i.managers.communication.GridIoManager.access$2100(Gri >>> dIoManager.java:108) >>> at o.a.i.i.managers.communication.GridIoManager$6.run(GridIoMan >>> ager.java:783) >>> at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java >>> :428) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> We encounter that log time to time specially when loading big >>> partitioned caches across the cluster. What does stripes threads do? Is >>> there a way to avoid that? >>> >>> Regards. >>> >>> -- >>> Alper Tekinalp >>> >>> Software Developer >>> Evam Streaming Analytics >>> >>> Atatürk Mah. Turgut Özal Bulv. >>> Gardenya 5 Plaza K:6 Ataşehir >>> 34758 İSTANBUL >>> >>> Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 >>> www.evam.com.tr >>> <http://www.evam.com> >>> >> >> >> >> -- >> Best regards, >> Andrey V. Mashenkov >> > > > > -- > Alper Tekinalp > > Software Developer > Evam Streaming Analytics > > Atatürk Mah. Turgut Özal Bulv. > Gardenya 5 Plaza K:6 Ataşehir > 34758 İSTANBUL > > Tel: +90 216 455 01 53 Fax: +90 216 455 01 54 > www.evam.com.tr > <http://www.evam.com> > -- Best regards, Andrey V. Mashenkov
