Hi, We are trying to convert our app to use Ignite in place of our current caching implementation(ifinispan). It will work fine for an hour or so, and then things begin to fall apart. Eventually all our applications AJP threads will be blocked waiting on ignite to retrieve cached objects.
We are currently using 2.2.
Here are some of the log messages we see as things begin to deterioriate:
[CLIENT] WARNING [org.apache.ignite.internal.diagnostic]
(grid-timeout-worker-#15%null%) Found long running cache future
[startTime=10:09:00.032, curTime=10:10:45.486,
fut=GridPartitionedSingleGetFuture [topVer=AffinityTopologyVersion [topVer=5,
minorTopVer=0], key=UserKeyCacheObjectImpl [part=75, val=bisyspa,
hasValBytes=true], readThrough=true, forcePrimary=false,
futId=b382ea9b461-fd143a5a-1e15-4157-ab2c-e1b11e84af1b, trackable=true,
subjId=9a152366-4292-459c-8959-403a4da42357, taskName=null,
deserializeBinary=true, skipVals=false, expiryPlc=null, canRemap=true,
needVer=false, keepCacheObjects=false, recovery=false, node=TcpDiscoveryNode
[id=bc32133d-4188-455f-8dbf-d4a13f9b4be6, addrs=[10.60.9.4, 127.0.0.1],
sockAddrs=[/127.0.0.1:47500, /10.60.9.4:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1532123004911, loc=false,
ver=2.2.0#20170915-sha1:5747ce6b, isClient=false]]]
[CLIENT] WARNING [org.apache.ignite.internal.util.typedef.G]
(grid-timeout-worker-#15%null%) >>> Possible starvation in striped pool.
Thread name: sys-stripe-1-#2%null%
Queue: []
Deadlock: false
Completed: 4
Thread [name="sys-stripe-1-#2%null%", id=130, state=RUNNABLE, blockCnt=0,
waitCnt=4]
at
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:247)
at
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:229)
at
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:280)
at
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:306)
at
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:229)
at
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:280)
at
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:229)
at
o.a.i.i.binary.BinaryObjectExImpl.appendValue(BinaryObjectExImpl.java:280)
at
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:229)
at
o.a.i.i.binary.BinaryObjectExImpl.toString(BinaryObjectExImpl.java:186)
at o.a.i.i.binary.BinaryObjectImpl.toString(BinaryObjectImpl.java:853)
at java.lang.String.valueOf(String.java:2849)
at o.a.i.i.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
o.a.i.i.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:884)
at
o.a.i.i.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:786)
at
o.a.i.i.processors.cache.distributed.near.GridNearSingleGetResponse.toString(GridNearSingleGetResponse.java:317)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at
o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:336)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1400(GridDhtAtomicCache.java:129)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:421)
at
o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:416)
at
o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
at
o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
at
o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
at
o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at
o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:748)
[SERVER] WARNING: Communication SPI session write timed out (consider
increasing 'socketWriteTimeout' configuration property)
[remoteAddr=as0418alyn.dmz.genworth.net/172.16.32.171:47100, writeTimeout=30000]
I don't have a thread dump at the moment of that stripe error, but have
attached what I do have that shows many of the threads waiting on a get. Also
attached are our ignite server and client configs. We are trying to run with 2
servers and 2 clients; I have attached one client(the other didn't show
anything happening at all), and both server thread dumps.
Thanks in advance for any insight.
<<attachment: dump_configs.zip>>
