We're using ignite in a 3 node grid with SSL just hit an issue where after
a period of time (hours after starting), 2 of the 3 nodes seem to have lost
connectivity and we see the following stack trace over and over.
The cluster starts up fine so I doubt it's an issue with the certificates
or keystores. Also bouncing the ignite instances seems to have "fixed"
it. Any ideas as to what could have happened?
Thanks,
Bryan
2018-02-27 14:52:36,071 INFO [grid-nio-worker-tcp-comm-2-#27]
o.a.i.s.c.tcp.TcpCommunicationSpi - Accepted incoming communication
connection [locAddr=/100.96.3.72:47100, rmtAddr=/100.96.6.183:45484]
2018-02-27 14:52:37,072 ERROR [grid-nio-worker-tcp-comm-2-#27]
o.a.i.s.c.tcp.TcpCommunicationSpi - Failed to process selector key
[ses=GridSelectorNioSessionImpl [worker=DirectNioClientWorker
[super=AbstractNioClientWorker [idx=2, bytesRcvd=17479234, bytesSent=0,
bytesRcvd0=2536, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-tcp-comm-2, igniteInstanceName=null, finished=false,
hashCode=1854311052, interrupted=false,
runner=grid-nio-worker-tcp-comm-2-#27]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/
100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030,
closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448,
bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071,
lastRcvTime=1519743156071, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@497350a6, directMode=true],
GridConnectionBytesVerifyFilter, SSL filter], accepted=true]]]
javax.net.ssl.SSLException: Failed to encrypt data (SSL engine error)
[status=CLOSED, handshakeStatus=NEED_UNWRAP, ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2,
bytesRcvd=17479234, bytesSent=0, bytesRcvd0=2536, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2,
igniteInstanceName=null, finished=false, hashCode=1854311052,
interrupted=false, runner=grid-nio-worker-tcp-comm-2-#27]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/
100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030,
closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448,
bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071,
lastRcvTime=1519743156071, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=org.apache.ignite.internal.util.nio.GridDirectParser@497350a6,
directMode=true], GridConnectionBytesVerifyFilter, SSL filter],
accepted=true]]]
at
org.apache.ignite.internal.util.nio.ssl.GridNioSslHandler.encrypt(GridNioSslHandler.java:379)
at
org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.encrypt(GridNioSslFilter.java:270)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWriteSsl(GridNioServer.java:1418)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processWrite(GridNioServer.java:1287)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2275)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2048)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1717)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
2018-02-27 14:52:37,072 WARN [grid-nio-worker-tcp-comm-2-#27]
o.a.i.s.c.tcp.TcpCommunicationSpi - Closing NIO session because of
unhandled exception [cls=class o.a.i.i.util.nio.GridNioException,
msg=Failed to encrypt data (SSL engine error) [status=CLOSED,
handshakeStatus=NEED_UNWRAP, ses=GridSelectorNioSessionImpl
[worker=DirectNioClientWorker [super=AbstractNioClientWorker [idx=2,
bytesRcvd=17479234, bytesSent=0, bytesRcvd0=2536, bytesSent0=0,
select=true, super=GridWorker [name=grid-nio-worker-tcp-comm-2,
igniteInstanceName=null, finished=false, hashCode=1854311052,
interrupted=false, runner=grid-nio-worker-tcp-comm-2-#27]]],
writeBuf=java.nio.DirectByteBuffer[pos=0 lim=10 cap=32768],
readBuf=java.nio.DirectByteBuffer[pos=0 lim=32768 cap=32768],
inRecovery=null, outRecovery=null, super=GridNioSessionImpl [locAddr=/
100.96.3.72:47100, rmtAddr=/100.96.6.183:45484, createTime=1519743156030,
closeTime=0, bytesSent=2448, bytesRcvd=2536, bytesSent0=2448,
bytesRcvd0=2536, sndSchedTime=1519743156071, lastSndTime=1519743156071,
lastRcvTime=1519743156071, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter
[parser=o.a.i.i.util.nio.GridDirectParser@497350a6, directMode=true],
GridConnectionBytesVerifyFilter, SSL filter], accepted=true]]]]
2018-02-27 14:52:37,321 INFO [grid-nio-worker-tcp-comm-3-#28]
o.a.i.s.c.tcp.TcpCommunicationSpi - Accepted incoming communication
connection [locAddr=/100.96.3.72:47100, rmtAddr=/100.96.6.183:45490]
