Hello, We have a 36 Node Ignite Cluster in Production taking a relatively high volume of Requests (6-10M Requests/hour) In general, it behaves beautifully. But we see occasional Spikes in response time (1-2 per hour - affecting 1-6 Nodes) Overall, these affect a handful of Requests (8-10 Requests) But still, they are annoying to my downstream Clients.
In many of these Spikes, we see the following ERROR: ``` [2018-03-30T05:30:09,690Z](grid-nio-worker-tcp-comm-4-#45)([]) ERROR - TcpCommunicationSpi - Failed to process selector key [ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=4, bytesRcvd=12231681761, bytesSent=1859740, bytesRcvd0=793212, bytesSent0=238, select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-4, igniteInstanceName=null, finished=false, hashCode=792369266, interrupted=false, runner=grid-nio-worker-tcp-comm-4-#45]]], writeBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768], inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/172.17.0.2:47111, rmtAddr=/10.120.0.216:48046, createTime=1522387808681, closeTime=0, bytesSent=38, bytesRcvd=42, bytesSent0=38, bytesRcvd0=42, sndSchedTime=1522387808681, lastSndTime=1522387808681, lastRcvTime=1522387808681, readsPaused=false, filterChain=FilterChain[filters=[GridNioCodecFilter [parser=o.a.i.i.util.nio.GridDirectParser@792f8249, directMode=true], GridConnectionBytesVerifyFilter], accepted=true]]] ! java.io.IOException: Connection reset by peer ``` My question. Is this a symptom or a cause?? What exactly does this Exception mean?? My gut tells me that when I see these Spikes, Ignite may be doing something under the covers?? And the ` Failed to process selector key` is indicative of that… But I don't have enough information to isolate anything further The app sees a high enough volume that it is very difficult to get the logging tweaked to where we can find the needle in this haystack. In other words, we cannot really turn on DEBUG logging and such - unless it is _very_ targeted. Can anyone give me any further information that may help me figure out what is happening?? Is the `Failed to process selector key` just a symptom of a deeper problem?? Is there some configuration parameter that I can tweak to help Ignite not see these issues?? Thanks much, -- Chris -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
