Hi,I am testing large Ignite Cache of 900GB, on 4 node VM(96GB RAM, 8CPU and
500GB SAN Storage) Spark Ignite Cluster .It happened tow times after
reaching 350GB plus one or two nodes not processing data load and the data
load is stopped. Please advise, the CLuster , Server and Client Logs
below.Detailsvisor> topHosts:
4+===================================================================================================================================+|
Int./Ext. IPs | Node ID8(@) | Node Type | OS
| CPUs | MACs | CPU Load
|+===================================================================================================================================+|
0:0:0:0:0:0:0:1%lo | 1: F6605E96(@n1) | Server | Linux amd64
3.10.0-862.11.6.el7.x86_64 | 8 | FA:16:3E:52:96:C4 | 0.14 % ||
127.0.0.1 | 2: 2760B50C(@n11) | Client |
| | | || 64.102.213.190 | 3:
81855FF0(@n12) | Client | | |
|
|+--------------------+-------------------+-----------+----------------------------------------+------+-------------------+----------+|
0:0:0:0:0:0:0:1%lo | 1: 512609AB(@n0) | Server | Linux amd64
3.10.0-862.11.6.el7.x86_64 | 8 | FA:16:3E:E5:27:36 | 2.13 % ||
127.0.0.1 | 2: 72AA1490(@n5) | Client |
| | | || 64.102.212.151 | 3:
E218A964(@n6) | Client | | |
|
|+--------------------+-------------------+-----------+----------------------------------------+------+-------------------+----------+|
0:0:0:0:0:0:0:1%lo | 1: 4470553B(@n2) | Server | Linux amd64
3.10.0-862.11.6.el7.x86_64 | 8 | FA:16:3E:C4:F4:98 | 0.10 % ||
127.0.0.1 | 2: F0D1625A(@n7) | Client |
| | | || 64.102.213.13 | 3:
EF0C5A13(@n8) | Client | | |
|
|+--------------------+-------------------+-----------+----------------------------------------+------+-------------------+----------+|
0:0:0:0:0:0:0:1%lo | 1: F44497FE(@n3) | Server | Linux amd64
3.10.0-862.11.6.el7.x86_64 | 8 | FA:16:3E:26:72:FD | 0.21 % ||
127.0.0.1 | 2: DBA60939(@n4) | Client |
| | | || 64.102.213.220 | 3:
65FA421F(@n9) | Client | | |
| || | 4: 8CBFE426(@n10) | Client |
| | |
|+-----------------------------------------------------------------------------------------------------------------------------------+Summary:+--------------------------------------+|
Active | true || Total hosts | 4
|| Total nodes | 13 || Total CPUs | 32
|| Avg. CPU load | 0.61 % || Avg. free heap | 71.00 %
|| Avg. Up time | 30:22:52 || Snapshot time | 2018-10-08
14:19:47 |+--------------------------------------+visor> nodeSelect node
from:+==========================================================================================+|
# | Node ID8(@), IP | Node Type | Up Time | CPUs | CPU Load
| Free Heap
|+==========================================================================================+|
0 | 512609AB(@n0), 64.102.212.151 | Server | 30:23:14 | 8 | 4.33 %
| 36.00 % || 1 | F6605E96(@n1), 64.102.213.190 | Server | 30:23:10 |
8 | 0.90 % | 56.00 % || 2 | 4470553B(@n2), 64.102.213.13 | Server
| 30:23:07 | 8 | 0.20 % | 78.00 % || 3 | F44497FE(@n3),
64.102.213.220 | Server | 30:23:03 | 8 | 0.17 % | 44.00 % || 4 |
DBA60939(@n4), 64.102.213.220 | Client | 14:21:12 | 8 | 0.17 % |
66.00 % || 5 | 72AA1490(@n5), 64.102.212.151 | Client | 14:21:06 | 8
| 0.17 % | 78.00 % || 6 | E218A964(@n6), 64.102.212.151 | Client |
14:21:07 | 8 | 0.17 % | 71.00 % || 7 | F0D1625A(@n7), 64.102.213.13
| Client | 14:21:06 | 8 | 0.07 % | 84.00 % || 8 | EF0C5A13(@n8),
64.102.213.13 | Client | 14:21:06 | 8 | 0.07 % | 83.00 % || 9 |
65FA421F(@n9), 64.102.213.220 | Client | 14:21:07 | 8 | 0.10 % |
64.00 % || 10 | 8CBFE426(@n10), 64.102.213.220 | Client | 14:21:06 | 8
| 0.13 % | 76.00 % || 11 | 2760B50C(@n11), 64.102.213.190 | Client |
14:21:07 | 8 | 0.13 % | 78.00 % || 12 | 81855FF0(@n12),
64.102.213.190 | Client | 14:21:06 | 8 | 0.10 % | 81.00 %
|+------------------------------------------------------------------------------------------+*Server
Log Message:*[11:59:15] ^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:32] Topology snapshot [ver=114, servers=4,
clients=2, CPUs=32, offheap=480.0GB, heap=28.0GB][11:59:32] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:32]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:32] Data Regions
Configured:[11:59:32] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:32] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:32]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:32] Topology snapshot [ver=115, servers=4,
clients=3, CPUs=32, offheap=560.0GB, heap=35.0GB][11:59:32] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:32]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:32] Data Regions
Configured:[11:59:32] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:32] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:32]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:33] Topology snapshot [ver=116, servers=4,
clients=4, CPUs=32, offheap=640.0GB, heap=42.0GB][11:59:33] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:33]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:33] Data Regions
Configured:[11:59:33] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:33] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:33]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:33] Topology snapshot [ver=117, servers=4,
clients=5, CPUs=32, offheap=720.0GB, heap=49.0GB][11:59:33] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:33]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:33] Data Regions
Configured:[11:59:33] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:33] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:33]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:34] Topology snapshot [ver=118, servers=4,
clients=6, CPUs=32, offheap=800.0GB, heap=57.0GB][11:59:34] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:34]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:34] Data Regions
Configured:[11:59:34] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:34] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:34]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:34] Topology snapshot [ver=119, servers=4,
clients=7, CPUs=32, offheap=880.0GB, heap=64.0GB][11:59:34] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:34]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:34] Data Regions
Configured:[11:59:34] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:34] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:34]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:34] Topology snapshot [ver=120, servers=4,
clients=8, CPUs=32, offheap=960.0GB, heap=71.0GB][11:59:34] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:34]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:34] Data Regions
Configured:[11:59:34] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:34] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:34]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][11:59:34] Topology snapshot [ver=121, servers=4,
clients=9, CPUs=32, offheap=1000.0GB, heap=78.0GB][11:59:34] ^-- Node
[id=F6605E96-47C9-479B-A840-03316500C9A3, clusterState=ACTIVE][11:59:34]
^-- Baseline [id=0, size=4, online=4, offline=0][11:59:34] Data Regions
Configured:[11:59:34] ^-- default_mem_region [initSize=256.0 MiB,
maxSize=20.0 GiB, persistenceEnabled=true][11:59:34] ^-- q_major
[initSize=10.0 GiB, maxSize=30.0 GiB, persistenceEnabled=true][11:59:34]
^-- q_minor [initSize=10.0 GiB, maxSize=30.0 GiB,
persistenceEnabled=true][14:33:15,872][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-3, igniteInstanceName=null,
finished=false, hashCode=254322881, interrupted=false,
runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.249.225:51449,
createTime=1538740798912, closeTime=0, bytesSent=397, bytesRcvd=302,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538742789216,
lastSndTime=1538742789216, lastRcvTime=1538742789216, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[21:43:26,312][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-0, igniteInstanceName=null,
finished=false, hashCode=2211598, interrupted=false,
runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.32.114:59525,
createTime=1538746249024, closeTime=0, bytesSent=2035, bytesRcvd=1532,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538767916701,
lastSndTime=1538767916701, lastRcvTime=1538767916701, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection timed out at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[23:02:32,031][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-1, igniteInstanceName=null,
finished=false, hashCode=1626735999, interrupted=false,
runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.249.225:51882,
createTime=1538769618223, closeTime=0, bytesSent=397, bytesRcvd=302,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538773344029,
lastSndTime=1538773344029, lastRcvTime=1538773344029, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[05:52:08,034][SEVERE][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=2, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-2, igniteInstanceName=null,
finished=false, hashCode=1810870884, interrupted=false,
runner=grid-nio-worker-client-listener-2-#32]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.177.186:54754,
createTime=1538797913271, closeTime=0, bytesSent=163, bytesRcvd=128,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538797924460,
lastSndTime=1538797924460, lastRcvTime=1538797924460, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[13:16:10,473][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-3, igniteInstanceName=null,
finished=false, hashCode=254322881, interrupted=false,
runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:60529,
createTime=1538824143152, closeTime=0, bytesSent=280, bytesRcvd=215,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538824568991,
lastSndTime=1538824568991, lastRcvTime=1538824568991, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[16:43:22,848][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-0, igniteInstanceName=null,
finished=false, hashCode=2211598, interrupted=false,
runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:57693,
createTime=1538836966711, closeTime=0, bytesSent=163, bytesRcvd=128,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538837001780,
lastSndTime=1538837001780, lastRcvTime=1538837001780, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[19:11:56,770][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-1, igniteInstanceName=null,
finished=false, hashCode=1626735999, interrupted=false,
runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:49209,
createTime=1538845894215, closeTime=0, bytesSent=163, bytesRcvd=128,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538845911149,
lastSndTime=1538845911149, lastRcvTime=1538845911149, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[21:32:26,339][SEVERE][grid-nio-worker-client-listener-2-#32][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=2, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-2, igniteInstanceName=null,
finished=false, hashCode=1810870884, interrupted=false,
runner=grid-nio-worker-client-listener-2-#32]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:65323,
createTime=1538852004067, closeTime=0, bytesSent=280, bytesRcvd=215,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538854342759,
lastSndTime=1538854342759, lastRcvTime=1538854342759, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[09:27:11,456][SEVERE][grid-nio-worker-client-listener-3-#33][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=3, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-3, igniteInstanceName=null,
finished=false, hashCode=254322881, interrupted=false,
runner=grid-nio-worker-client-listener-3-#33]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:53182,
createTime=1538897206161, closeTime=0, bytesSent=163, bytesRcvd=128,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538897228457,
lastSndTime=1538897228457, lastRcvTime=1538897228457, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[16:27:51,008][SEVERE][grid-nio-worker-client-listener-0-#30][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=0, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-0, igniteInstanceName=null,
finished=false, hashCode=2211598, interrupted=false,
runner=grid-nio-worker-client-listener-0-#30]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:58799,
createTime=1538920292729, closeTime=0, bytesSent=397, bytesRcvd=302,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538922468102,
lastSndTime=1538922468102, lastRcvTime=1538922468102, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at
java.lang.Thread.run(Thread.java:748)[23:43:07,105][SEVERE][grid-nio-worker-client-listener-1-#31][ClientListenerProcessor]
Failed to process selector key [ses=GridSelectorNioSessionImpl
[worker=ByteBufferNioClientWorker [readBuf=java.nio.HeapByteBuffer[pos=0
lim=8192 cap=8192], super=AbstractNioClientWorker [idx=1, bytesRcvd=0,
bytesSent=0, bytesRcvd0=0, bytesSent0=0, select=true, super=GridWorker
[name=grid-nio-worker-client-listener-1, igniteInstanceName=null,
finished=false, hashCode=1626735999, interrupted=false,
runner=grid-nio-worker-client-listener-1-#31]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, super=GridNioSessionImpl
[locAddr=/64.102.213.190:10800, rmtAddr=/10.82.224.11:57332,
createTime=1538947042237, closeTime=0, bytesSent=631, bytesRcvd=476,
bytesSent0=0, bytesRcvd0=0, sndSchedTime=1538948581568,
lastSndTime=1538948581568, lastRcvTime=1538948581568, readsPaused=false,
filterChain=FilterChain[filters=[GridNioAsyncNotifyFilter,
GridNioCodecFilter [parser=ClientListenerBufferedParser, directMode=false]],
accepted=true]]]java.io.IOException: Connection reset by peer at
sun.nio.ch.FileDispatcherImpl.read0(Native Method) at
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at
sun.nio.ch.IOUtil.read(IOUtil.java:197) at
sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at
org.apache.ignite.internal.util.nio.GridNioServer$ByteBufferNioClientWorker.processRead(GridNioServer.java:1085)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2339)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2110)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1764)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)*Client Log Message:*2018-10-08
14:03:06 INFO IgniteKernal:566 - Metrics for local node (to disable set
'metricsLogFrequency' to 0) ^-- Node [id=2760b50c, uptime=14:03:31.475]
^-- H/N/C [hosts=4, nodes=13, CPUs=32] ^-- CPU [cur=0.07%, avg=0.1%,
GC=0%] ^-- PageMemory [pages=0] ^-- Heap [used=2279MB, free=68.69%,
comm=3143MB] ^-- Non heap [used=138MB, free=-1%, comm=142MB] ^--
Outbound messages queue [size=0] ^-- Public thread pool [active=0,
idle=0, qSize=0] ^-- System thread pool [active=3, idle=0,
qSize=0]2018-10-08 14:03:55 WARN diagnostic:571 - Failed to wait for
partition map exchange [topVer=AffinityTopologyVersion [topVer=123,
minorTopVer=0], node=2760b50c-0617-4dfd-bbba-23e842b362f5]. Consider
changing TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to
non default value to avoid this message. Dumping pending objects that might
be the cause: 2018-10-08 14:03:55 WARN diagnostic:571 - Ready affinity
version: AffinityTopologyVersion [topVer=122, minorTopVer=0]2018-10-08
14:03:55 WARN diagnostic:571 - Last exchange future:
GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=123,
nodeId8=2760b50c, msg=Node left: TcpDiscoveryNode
[id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_LEFT,
tstamp=1538931072415], crd=TcpDiscoveryNode
[id=512609ab-1fcf-4a51-bb7c-3965abc6b386, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.212.151],
sockAddrs=[ccrc-rptignite-stg1-01.cisco.com/64.102.212.151:47500,
/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1538740774375, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false],
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=123, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=80d74493-b1c2-48ee-a998-98a927692a0d,
addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], topVer=123,
nodeId8=2760b50c, msg=Node left: TcpDiscoveryNode
[id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], type=NODE_LEFT,
tstamp=1538931072415], nodeId=80d74493, evt=NODE_LEFT], added=true,
initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=true,
hash=1553040840], init=true, lastVer=null, partReleaseFut=null,
exchActions=null, affChangeMsg=null, initTs=1538931072455,
centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null,
done=false, state=CLIENT, evtLatch=0,
remaining=[f44497fe-3f02-453d-8407-078807e74288,
f6605e96-47c9-479b-a840-03316500c9a3, 512609ab-1fcf-4a51-bb7c-3965abc6b386,
4470553b-4f25-48cc-abb6-ac260f4d6301], super=GridFutureAdapter
[ignoreInterrupts=false, state=INIT, res=null, hash=992063964]]2018-10-08
14:03:55 WARN GridCachePartitionExchangeManager:571 - First 10 pending
exchange futures [total=2]2018-10-08 14:03:55 WARN diagnostic:571 - Last 10
exchange futures (total: 4):2018-10-08 14:03:55 WARN diagnostic:571 - >>>
GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=123,
minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode
[id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=false]2018-10-08
14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture
[topVer=AffinityTopologyVersion [topVer=122, minorTopVer=0],
evt=NODE_JOINED, evtNode=TcpDiscoveryNode
[id=80d74493-b1c2-48ee-a998-98a927692a0d, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:47501,
/0:0:0:0:0:0:0:1%lo:47501, /127.0.0.1:47501], discPort=47501, order=122,
intOrder=68, lastExchangeTime=1538930290162, loc=false,
ver=2.6.0#20180710-sha1:669feacc, isClient=false], done=true]2018-10-08
14:03:55 WARN diagnostic:571 - >>> GridDhtPartitionsExchangeFuture
[topVer=AffinityTopologyVersion [topVer=121, minorTopVer=0],
evt=NODE_JOINED, evtNode=TcpDiscoveryNode
[id=81855ff0-46db-4284-ade4-3823667cc194, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=121, intOrder=67,
lastExchangeTime=1538740774476, loc=false, ver=2.6.0#20180710-sha1:669feacc,
isClient=true], done=true]2018-10-08 14:03:55 WARN diagnostic:571 - >>>
GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=120,
minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode
[id=2760b50c-0617-4dfd-bbba-23e842b362f5, addrs=[0:0:0:0:0:0:0:1%lo,
127.0.0.1, 64.102.213.190],
sockAddrs=[host-64-102-213-190.cisco.com/64.102.213.190:0,
/0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0], discPort=0, order=120, intOrder=0,
lastExchangeTime=1538740772995, loc=true, ver=2.6.0#20180710-sha1:669feacc,
isClient=true], done=true]2018-10-08 14:03:55 WARN diagnostic:571 - Latch
manager state: ExchangeLatchManager [serverLatches={},
clientLatches={}]2018-10-08 14:03:55 WARN diagnostic:571 - Pending
transactions:2018-10-08 14:03:55 WARN diagnostic:571 - Pending explicit
locks:2018-10-08 14:03:55 WARN diagnostic:571 - Pending cache
futures:2018-10-08 14:03:55 WARN diagnostic:571 - Pending atomic cache
futures:2018-10-08 14:03:55 WARN diagnostic:571 - Pending data streamer
futures:2018-10-08 14:03:55 WARN diagnostic:571 - Pending transaction
deadlock detection futures:2018-10-08 14:04:06 INFO IgniteKernal:566 -
Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^--
Node [id=2760b50c, uptime=14:04:31.475] ^-- H/N/C [hosts=4, nodes=13,
CPUs=32] ^-- CPU [cur=0.13%, avg=0.1%, GC=0%] ^-- PageMemory [pages=0]
^-- Heap [used=2295MB, free=68.48%, comm=3143MB] ^-- Non heap
[used=138MB, free=-1%, comm=142MB] ^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool
[active=3, idle=0, qSize=0]2018-10-08 14:05:06 INFO IgniteKernal:566 -
Thanks
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
