[
https://issues.apache.org/jira/browse/IGNITE-9184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17023160#comment-17023160
]
Dmitriy Sorokin commented on IGNITE-9184:
-----------------------------------------
[~mmuzaf]
I wrote the new version of reproducer (see the StressTest3.java attachment)
which provides true concurrent nodes restart, but subject problem with cluster
hanging never met, neither on version 2.6 nor current master branch. However,
short-time hangings with logs similar to ones published at comment above still
appearing at the current master:
{noformat}
[2019-12-06 13:10:21,381][INFO
][exchange-worker-#1448818%hang.StressTest325879%][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=51762, minorTopVer=0], crd=false,
evt=NODE_JOINED, evtNode=92f30d54-e3ec-4292-84fa-e1f399325881, customEvt=null,
allowMerge=true]
[2019-12-06 13:10:21,381][INFO
][exchange-worker-#1448818%hang.StressTest325879%][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future [topVer=AffinityTopologyVersion
[topVer=51762, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED]
[2019-12-06 13:10:21,381][INFO
][exchange-worker-#1448936%hang.StressTest325881%][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=51762, minorTopVer=0], crd=false,
evt=NODE_JOINED, evtNode=92f30d54-e3ec-4292-84fa-e1f399325881, customEvt=null,
allowMerge=true]
[2019-12-06 13:10:21,382][INFO
][exchange-worker-#1448936%hang.StressTest325881%][IgniteCacheDatabaseSharedManager]
Configured data regions started successfully [total=3]
[2019-12-06 13:10:21,382][INFO
][exchange-worker-#1448936%hang.StressTest325881%][msg] Components activation
performed in 1 ms.
[2019-12-06 13:10:21,383][INFO
][sys-#1448940%hang.StressTest325881%][GridCacheProcessor] Started cache
[name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc,
mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false]
[2019-12-06 13:10:21,383][INFO
][exchange-worker-#1448936%hang.StressTest325881%][GridCacheProcessor] Started
cache [name=default, id=1544803905, dataRegionName=default, mode=PARTITIONED,
atomicity=ATOMIC, backups=1, mvcc=false]
[2019-12-06 13:10:21,383][INFO
][exchange-worker-#1448936%hang.StressTest325881%][GridCacheProcessor] Starting
caches on local join performed in 10 ms.
[2019-12-06 13:10:21,381][WARN ][Thread-9][StressTest325881] Nodes started on
local machine require more than 80% of physical RAM what can lead to
significant slowdown due to swapping (please decrease JVM heap size, data
region size or checkpoint buffer size) [required=35144MB, available=16280MB]
[2019-12-06 13:10:21,384][INFO
][exchange-worker-#1448936%hang.StressTest325881%][GridDhtPartitionsExchangeFuture]
Skipped waiting for partitions release future (local node is joining)
[topVer=AffinityTopologyVersion [topVer=51762, minorTopVer=0]]
[2019-12-06 13:10:21,385][INFO
][grid-nio-worker-tcp-comm-2-#1448576%hang.StressTest325875%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45014,
rmtAddr=/127.0.0.1:55957]
[2019-12-06 13:10:21,386][INFO
][grid-nio-worker-tcp-comm-0-#1448902%hang.StressTest325881%][TcpCommunicationSpi]
Established outgoing communication connection [locAddr=/127.0.0.1:55957,
rmtAddr=/127.0.0.1:45014]
[2019-12-06 13:10:21,386][INFO
][exchange-worker-#1448936%hang.StressTest325881%][time] Finished exchange init
[topVer=AffinityTopologyVersion [topVer=51762, minorTopVer=0], crd=false]
[2019-12-06 13:10:21,413][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55958]
[2019-12-06 13:10:21,463][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55959]
[2019-12-06 13:10:21,514][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55960]
[2019-12-06 13:10:21,565][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55961]
[2019-12-06 13:10:21,569][INFO
][sys-#1448612%hang.StressTest325875%][GridCachePartitionExchangeManager]
Ignore single message without exchange id (there is exchange in progress)
[nodeId=4422f968-e654-4376-bef6-262125325879]
[2019-12-06 13:10:21,617][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55962]
[2019-12-06 13:10:21,667][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55963]
[2019-12-06 13:10:21,719][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55964]
[2019-12-06 13:10:21,771][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55965]
[2019-12-06 13:10:21,821][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55966]
[2019-12-06 13:10:21,873][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55967]
[2019-12-06 13:10:21,923][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55968]
[2019-12-06 13:10:21,974][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55969]
[2019-12-06 13:10:22,025][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55970]
[2019-12-06 13:10:22,077][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55971]
[2019-12-06 13:10:22,128][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55972]
[2019-12-06 13:10:22,181][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55973]
[2019-12-06 13:10:22,233][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55974]
[2019-12-06 13:10:22,284][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55975]
[2019-12-06 13:10:22,318][INFO
][sys-#1448608%hang.StressTest325875%][GridCachePartitionExchangeManager]
Ignore single message without exchange id (there is exchange in progress)
[nodeId=8b378ee1-fe77-413a-95f2-6f3847f25877]
[2019-12-06 13:10:22,319][INFO
][sys-#1448718%hang.StressTest325877%][GridCachePartitionExchangeManager] Delay
process full message without exchange id (there is exchange in progress)
[nodeId=762756fa-f39c-4d0d-ae39-ab24f0525875]
[2019-12-06 13:10:22,319][INFO
][sys-#1448827%hang.StressTest325879%][GridCachePartitionExchangeManager] Delay
process full message without exchange id (there is exchange in progress)
[nodeId=762756fa-f39c-4d0d-ae39-ab24f0525875]
[2019-12-06 13:10:22,335][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55976]
[2019-12-06 13:10:22,387][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55977]
[2019-12-06 13:10:22,438][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55978]
[2019-12-06 13:10:22,490][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55980]
[2019-12-06 13:10:22,540][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55982]
[2019-12-06 13:10:22,592][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55983]
[2019-12-06 13:10:22,643][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55984]
[2019-12-06 13:10:22,695][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55985]
[2019-12-06 13:10:22,746][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55986]
[2019-12-06 13:10:22,797][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55987]
[2019-12-06 13:10:22,847][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55988]
[2019-12-06 13:10:22,899][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55989]
[2019-12-06 13:10:22,950][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55990]
[2019-12-06 13:10:23,000][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55991]
[2019-12-06 13:10:23,051][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55992]
[2019-12-06 13:10:23,103][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55993]
[2019-12-06 13:10:23,155][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55994]
[2019-12-06 13:10:23,206][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55995]
[2019-12-06 13:10:23,257][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55996]
[2019-12-06 13:10:23,308][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55997]
[2019-12-06 13:10:23,359][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55998]
[2019-12-06 13:10:23,411][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:55999]
[2019-12-06 13:10:23,462][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56000]
[2019-12-06 13:10:23,513][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56001]
[2019-12-06 13:10:23,564][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56002]
[2019-12-06 13:10:23,615][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56003]
[2019-12-06 13:10:23,665][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56004]
[2019-12-06 13:10:23,716][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56005]
[2019-12-06 13:10:23,767][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56006]
[2019-12-06 13:10:23,819][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56007]
[2019-12-06 13:10:23,871][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56008]
[2019-12-06 13:10:23,923][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56009]
[2019-12-06 13:10:23,973][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56010]
[2019-12-06 13:10:24,025][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56011]
[2019-12-06 13:10:24,078][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56012]
[2019-12-06 13:10:24,130][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56013]
[2019-12-06 13:10:24,181][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56014]
[2019-12-06 13:10:24,233][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56015]
[2019-12-06 13:10:24,285][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56016]
[2019-12-06 13:10:24,338][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56017]
[2019-12-06 13:10:24,389][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56018]
[2019-12-06 13:10:24,440][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56019]
[2019-12-06 13:10:24,491][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56020]
[2019-12-06 13:10:24,541][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56021]
[2019-12-06 13:10:24,591][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56022]
[2019-12-06 13:10:24,642][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56023]
[2019-12-06 13:10:24,694][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56024]
[2019-12-06 13:10:24,746][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56025]
[2019-12-06 13:10:24,798][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56026]
[2019-12-06 13:10:24,851][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56027]
[2019-12-06 13:10:24,902][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56028]
[2019-12-06 13:10:24,954][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56029]
[2019-12-06 13:10:25,005][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56030]
[2019-12-06 13:10:25,058][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56031]
[2019-12-06 13:10:25,109][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56032]
[2019-12-06 13:10:25,160][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56033]
[2019-12-06 13:10:25,210][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56034]
[2019-12-06 13:10:25,262][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56035]
[2019-12-06 13:10:25,313][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56036]
[2019-12-06 13:10:25,365][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56037]
[2019-12-06 13:10:25,417][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56038]
[2019-12-06 13:10:25,469][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56039]
[2019-12-06 13:10:25,521][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56041]
[2019-12-06 13:10:25,572][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56042]
[2019-12-06 13:10:25,624][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56043]
[2019-12-06 13:10:25,676][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56044]
[2019-12-06 13:10:25,728][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56045]
[2019-12-06 13:10:25,781][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56046]
[2019-12-06 13:10:25,833][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56047]
[2019-12-06 13:10:25,884][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56048]
[2019-12-06 13:10:25,935][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56049]
[2019-12-06 13:10:25,986][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56050]
[2019-12-06 13:10:26,039][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56051]
[2019-12-06 13:10:26,090][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56052]
[2019-12-06 13:10:26,141][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56053]
[2019-12-06 13:10:26,191][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56054]
[2019-12-06 13:10:26,244][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56055]
[2019-12-06 13:10:26,297][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56056]
[2019-12-06 13:10:26,347][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56057]
[2019-12-06 13:10:26,399][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56058]
[2019-12-06 13:10:26,451][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56059]
[2019-12-06 13:10:26,504][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56060]
[2019-12-06 13:10:26,555][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56061]
[2019-12-06 13:10:26,606][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56062]
[2019-12-06 13:10:26,658][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56063]
[2019-12-06 13:10:26,709][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56064]
[2019-12-06 13:10:26,762][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56065]
[2019-12-06 13:10:26,813][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56066]
[2019-12-06 13:10:26,865][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56067]
[2019-12-06 13:10:26,917][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56068]
[2019-12-06 13:10:26,970][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56069]
[2019-12-06 13:10:27,023][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56070]
[2019-12-06 13:10:27,074][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56071]
[2019-12-06 13:10:27,126][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56072]
[2019-12-06 13:10:27,178][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56073]
[2019-12-06 13:10:27,229][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56074]
[2019-12-06 13:10:27,280][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56075]
[2019-12-06 13:10:27,331][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56076]
[2019-12-06 13:10:27,382][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56077]
[2019-12-06 13:10:27,433][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56078]
[2019-12-06 13:10:27,483][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56079]
[2019-12-06 13:10:27,534][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56080]
[2019-12-06 13:10:27,585][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56081]
[2019-12-06 13:10:27,638][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56082]
[2019-12-06 13:10:27,691][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56083]
[2019-12-06 13:10:27,743][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56084]
[2019-12-06 13:10:27,794][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56085]
[2019-12-06 13:10:27,844][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56086]
[2019-12-06 13:10:27,896][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56087]
[2019-12-06 13:10:27,947][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56088]
[2019-12-06 13:10:27,999][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56089]
[2019-12-06 13:10:28,050][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56090]
[2019-12-06 13:10:28,100][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56091]
[2019-12-06 13:10:28,151][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56092]
[2019-12-06 13:10:28,204][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56093]
[2019-12-06 13:10:28,255][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56094]
[2019-12-06 13:10:28,305][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56095]
[2019-12-06 13:10:28,358][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56096]
[2019-12-06 13:10:28,411][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56097]
[2019-12-06 13:10:28,462][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56098]
[2019-12-06 13:10:28,512][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56099]
[2019-12-06 13:10:28,563][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56100]
[2019-12-06 13:10:28,616][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56101]
[2019-12-06 13:10:28,668][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56102]
[2019-12-06 13:10:28,721][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56103]
[2019-12-06 13:10:28,772][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56104]
[2019-12-06 13:10:28,824][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56105]
[2019-12-06 13:10:28,876][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56106]
[2019-12-06 13:10:28,928][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56107]
[2019-12-06 13:10:28,979][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56108]
[2019-12-06 13:10:29,032][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56110]
[2019-12-06 13:10:29,084][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56111]
[2019-12-06 13:10:29,137][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56112]
[2019-12-06 13:10:29,189][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56113]
[2019-12-06 13:10:29,240][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56114]
[2019-12-06 13:10:29,291][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56115]
[2019-12-06 13:10:29,343][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56116]
[2019-12-06 13:10:29,396][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56117]
[2019-12-06 13:10:29,447][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56118]
[2019-12-06 13:10:29,498][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56119]
[2019-12-06 13:10:29,550][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56120]
[2019-12-06 13:10:29,602][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56121]
[2019-12-06 13:10:29,655][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56122]
[2019-12-06 13:10:29,705][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56123]
[2019-12-06 13:10:29,756][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56124]
[2019-12-06 13:10:29,807][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56125]
[2019-12-06 13:10:29,858][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56126]
[2019-12-06 13:10:29,909][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56127]
[2019-12-06 13:10:29,961][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56128]
[2019-12-06 13:10:30,013][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56129]
[2019-12-06 13:10:30,065][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56130]
[2019-12-06 13:10:30,116][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56131]
[2019-12-06 13:10:30,167][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56132]
[2019-12-06 13:10:30,218][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56133]
[2019-12-06 13:10:30,270][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56134]
[2019-12-06 13:10:30,322][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56135]
[2019-12-06 13:10:30,375][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56136]
[2019-12-06 13:10:30,427][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56137]
[2019-12-06 13:10:30,478][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56138]
[2019-12-06 13:10:30,530][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56139]
[2019-12-06 13:10:30,581][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56140]
[2019-12-06 13:10:30,632][INFO
][grid-nio-worker-tcp-comm-1-#1448909%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56141]
[2019-12-06 13:10:30,684][INFO
][grid-nio-worker-tcp-comm-2-#1448910%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56142]
[2019-12-06 13:10:30,735][INFO
][grid-nio-worker-tcp-comm-3-#1448911%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56143]
[2019-12-06 13:10:30,787][INFO
][grid-nio-worker-tcp-comm-0-#1448908%hang.StressTest325882%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/127.0.0.1:45013,
rmtAddr=/127.0.0.1:56144]
[2019-12-06 13:10:30,837][WARN
][exchange-worker-#1448606%hang.StressTest325875%][TcpCommunicationSpi]
Handshake timed out (will stop attempts to perform the handshake)
[node=61489ab7-4dab-40ec-a0a3-205457a25880,
connTimeoutStrategy=ExponentialBackoffTimeoutStrategy [maxTimeout=600000,
totalTimeout=10000, startNanos=1232774040710433, currTimeout=600000],
err=Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=1232774040710433,
currTimeout=600000]], addr=/127.0.0.1:45013,
failureDetectionTimeoutEnabled=true, timeout=0]
[2019-12-06
13:10:30,837][ERROR][exchange-worker-#1448606%hang.StressTest325875%][TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020698, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsFullMessage
[parts=HashMap {1544803905=GridDhtPartitionFullMap
{8b378ee1-fe77-413a-95f2-6f3847f25877=GridDhtPartitionMap [moving=97,
top=AffinityTopologyVersion [topVer=51758, minorTopVer=0], updateSeq=670,
size=748], 4422f968-e654-4376-bef6-262125325879=GridDhtPartitionMap
[moving=183, top=AffinityTopologyVersion [topVer=51758, minorTopVer=0],
updateSeq=529, size=706],
762756fa-f39c-4d0d-ae39-ab24f0525875=GridDhtPartitionMap [moving=84,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=788,
size=686]}, -2100569601=GridDhtPartitionFullMap
{8b378ee1-fe77-413a-95f2-6f3847f25877=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51758, minorTopVer=0], updateSeq=117,
size=100], 4422f968-e654-4376-bef6-262125325879=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51758, minorTopVer=0], updateSeq=106,
size=100], 762756fa-f39c-4d0d-ae39-ab24f0525875=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=128,
size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null,
partsToReload=null, topVer=AffinityTopologyVersion [topVer=51760,
minorTopVer=0], errs=null, resTopVer=null, partCnt=2,
super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null,
super=GridCacheMessage [msgId=550551, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=51760, minorTopVer=0],
err=null, skipPrepare=false]]]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is
node still alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of network
issues [nodeId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=[/127.0.0.1:45013]]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3621)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3385)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3125)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3008)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2848)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2807)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1373)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1297)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1242)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1258)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3276)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=1232774040710433,
currTimeout=600000]]
at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3506)
... 15 more
[2019-12-06 13:10:30,839][WARN ][tcp-disco-msg-worker-[8b378ee1 127.0.0.1:47500
crd]-#209160%hang.StressTest325875%][TcpDiscoverySpi] Failing client node due
to not receiving metrics updates from client node within
'IgniteConfiguration.clientFailureDetectionTimeout' (consider increasing
configuration property) [timeout=10000, node=TcpDiscoveryNode
[id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020698, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]]
[2019-12-06
13:10:30,837][ERROR][sys-#1448613%hang.StressTest325875%][TcpCommunicationSpi]
Failed to send message to remote node [node=TcpDiscoveryNode
[id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020698, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8,
ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsFullMessage
[parts=HashMap {1544803905=GridDhtPartitionFullMap
{8b378ee1-fe77-413a-95f2-6f3847f25877=GridDhtPartitionMap [moving=97,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=674,
size=656], 4422f968-e654-4376-bef6-262125325879=GridDhtPartitionMap
[moving=183, top=AffinityTopologyVersion [topVer=51758, minorTopVer=0],
updateSeq=529, size=706],
762756fa-f39c-4d0d-ae39-ab24f0525875=GridDhtPartitionMap [moving=84,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=788,
size=686]}, -2100569601=GridDhtPartitionFullMap
{8b378ee1-fe77-413a-95f2-6f3847f25877=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=119,
size=100], 4422f968-e654-4376-bef6-262125325879=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51758, minorTopVer=0], updateSeq=106,
size=100], 762756fa-f39c-4d0d-ae39-ab24f0525875=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=51760, minorTopVer=0], updateSeq=128,
size=100]}}, partCntrs=null, partCntrs2=null, partHistSuppliers=null,
partsToReload=null, topVer=AffinityTopologyVersion [topVer=51760,
minorTopVer=0], errs=null, resTopVer=null, partCnt=2,
super=GridDhtPartitionsAbstractMessage [exchId=null, lastVer=null,
super=GridCacheMessage [msgId=550564, depInfo=null,
lastAffChangedTopVer=AffinityTopologyVersion [topVer=51760, minorTopVer=0],
err=null, skipPrepare=false]]]]]
class org.apache.ignite.IgniteCheckedException: Failed to connect to node (is
node still alive?). Make sure that each ComputeTask and cache Transaction has a
timeout set in order to prevent parties from waiting forever in case of network
issues [nodeId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=[/127.0.0.1:45013]]
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3621)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3385)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createCommunicationClient(TcpCommunicationSpi.java:3125)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:3008)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2848)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2807)
at
org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:2031)
at
org.apache.ignite.internal.managers.communication.GridIoManager.sendToGridTopic(GridIoManager.java:2128)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.sendNoRetry(GridCacheIoManager.java:1373)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.sendAllPartitions(GridCachePartitionExchangeManager.java:1297)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1242)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.refreshPartitions(GridCachePartitionExchangeManager.java:1258)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3276)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.spi.IgniteSpiOperationTimeoutException:
Operation timed out [timeoutStrategy= ExponentialBackoffTimeoutStrategy
[maxTimeout=600000, totalTimeout=10000, startNanos=1232774040710433,
currTimeout=600000]]
at
org.apache.ignite.spi.ExponentialBackoffTimeoutStrategy.nextTimeout(ExponentialBackoffTimeoutStrategy.java:103)
at
org.apache.ignite.spi.TimeoutStrategy.nextTimeout(TimeoutStrategy.java:39)
at
org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioSession(TcpCommunicationSpi.java:3506)
... 15 more
[2019-12-06 13:10:30,843][WARN
][disco-event-worker-#1448712%hang.StressTest325877%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020708, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]
[2019-12-06 13:10:30,844][WARN
][disco-event-worker-#1448817%hang.StressTest325879%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020708, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]
[2019-12-06 13:10:30,842][WARN
][disco-event-worker-#1448926%hang.StressTest325882%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627021079, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]
[2019-12-06 13:10:30,842][WARN
][disco-event-worker-#1448605%hang.StressTest325875%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627020698, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]
[2019-12-06 13:10:30,843][WARN
][disco-event-worker-#1448935%hang.StressTest325881%][GridDiscoveryManager]
Node FAILED: TcpDiscoveryNode [id=61489ab7-4dab-40ec-a0a3-205457a25880,
consistentId=61489ab7-4dab-40ec-a0a3-205457a25880, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51759, intOrder=25882,
lastExchangeTime=1575627021169, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true]
[2019-12-06 13:10:30,843][INFO
][sys-#1448613%hang.StressTest325875%][GridCachePartitionExchangeManager]
Finished sending full message [msgTopVer=AffinityTopologyVersion [topVer=51760,
minorTopVer=0], groups=[CacheGroupContext [grp=default], CacheGroupContext
[grp=ignite-sys-cache]], latency=8526ms]
[2019-12-06 13:10:30,843][INFO
][disco-event-worker-#1448712%hang.StressTest325877%][GridDiscoveryManager]
Topology snapshot [ver=51763, locNode=8b378ee1, servers=4, clients=1,
state=ACTIVE, CPUs=8, offheap=3.2GB, heap=3.5GB]
[2019-12-06 13:10:30,843][INFO
][disco-event-worker-#1448712%hang.StressTest325877%][GridDiscoveryManager]
^-- Baseline [id=0, size=4, online=4, offline=0]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448817%hang.StressTest325879%][GridDiscoveryManager]
Topology snapshot [ver=51763, locNode=4422f968, servers=4, clients=1,
state=ACTIVE, CPUs=8, offheap=3.2GB, heap=3.5GB]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448817%hang.StressTest325879%][GridDiscoveryManager]
^-- Baseline [id=0, size=4, online=4, offline=0]
[2019-12-06 13:10:30,842][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridCachePartitionExchangeManager]
Finished sending full message [msgTopVer=AffinityTopologyVersion
[topVer=51760, minorTopVer=0], groups=[CacheGroupContext [grp=default],
CacheGroupContext [grp=ignite-sys-cache]], latency=10032ms]
[2019-12-06 13:10:30,844][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridCachePartitionExchangeManager]
Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion
[topVer=51760, minorTopVer=0], evt=NODE_LEFT,
node=b3675e70-d155-439f-b72a-a0c06af25873]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448605%hang.StressTest325875%][GridDiscoveryManager]
Topology snapshot [ver=51763, locNode=762756fa, servers=4, clients=1,
state=ACTIVE, CPUs=8, offheap=3.2GB, heap=3.5GB]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448605%hang.StressTest325875%][GridDiscoveryManager]
^-- Baseline [id=0, size=4, online=4, offline=0]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448926%hang.StressTest325882%][GridDiscoveryManager]
Topology snapshot [ver=51763, locNode=7de2c2a1, servers=4, clients=1,
state=ACTIVE, CPUs=8, offheap=0.1GB, heap=3.5GB]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448926%hang.StressTest325882%][GridDiscoveryManager]
^-- Baseline [id=0, size=4, online=4, offline=0]
[2019-12-06 13:10:30,844][INFO
][exchange-worker-#1448606%hang.StressTest325875%][time] Started exchange init
[topVer=AffinityTopologyVersion [topVer=51761, minorTopVer=0], crd=true,
evt=NODE_JOINED, evtNode=7de2c2a1-d747-4107-9544-a75e23e25882, customEvt=null,
allowMerge=true]
[2019-12-06 13:10:30,844][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=51761,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=51761, minorTopVer=0],
err=null]
[2019-12-06 13:10:30,844][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridDhtPartitionsExchangeFuture]
Completed partition exchange [localNode=762756fa-f39c-4d0d-ae39-ab24f0525875,
exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion
[topVer=51761, minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode
[id=7de2c2a1-d747-4107-9544-a75e23e25882,
consistentId=7de2c2a1-d747-4107-9544-a75e23e25882, addrs=ArrayList [127.0.0.1],
sockAddrs=HashSet [/127.0.0.1:0], discPort=0, order=51761, intOrder=25883,
lastExchangeTime=1575627021039, loc=false, ver=2.8.0#20191205-sha1:00000000,
isClient=true], done=true, newCrdFut=null], topVer=AffinityTopologyVersion
[topVer=51761, minorTopVer=0]]
[2019-12-06 13:10:30,844][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridDhtPartitionsExchangeFuture]
Exchange timings [startVer=AffinityTopologyVersion [topVer=51761,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=51761, minorTopVer=0],
stage="Waiting in exchange queue" (9785 ms), stage="Exchange parameters
initialization" (0 ms), stage="Determine exchange type" (0 ms), stage="Exchange
done" (0 ms), stage="Total time" (9785 ms)]
[2019-12-06 13:10:30,847][INFO
][exchange-worker-#1448606%hang.StressTest325875%][GridDhtPartitionsExchangeFuture]
Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=51761,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=51761, minorTopVer=0]]
[2019-12-06 13:10:30,844][INFO
][disco-event-worker-#1448935%hang.StressTest325881%][GridDiscoveryManager]
Topology snapshot [ver=51763, locNode=92f30d54, servers=4, clients=1,
state=ACTIVE, CPUs=8, offheap=3.2GB, heap=3.5GB]
{noformat}
> Cluster hangs during concurrent node client and server nodes restart
> --------------------------------------------------------------------
>
> Key: IGNITE-9184
> URL: https://issues.apache.org/jira/browse/IGNITE-9184
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.6
> Reporter: Mikhail Cherkasov
> Assignee: Dmitriy Sorokin
> Priority: Blocker
> Fix For: 2.8
>
> Attachments: StressTest2.java, StressTest3.java, logs, stacktrace
>
>
> Please check the attached test case and stack trace.
> I can see: "Failed to wait for initial partition map exchange" message.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
