[ https://issues.apache.org/jira/browse/IGNITE-10010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Muzafarov updated IGNITE-10010: ------------------------------------- Priority: Critical (was: Blocker) > Node halted if second node was stopped, then cache destroyed, then second > node returned > --------------------------------------------------------------------------------------- > > Key: IGNITE-10010 > URL: https://issues.apache.org/jira/browse/IGNITE-10010 > Project: Ignite > Issue Type: Bug > Affects Versions: 2.7 > Reporter: Sergey Kozlov > Assignee: Alexey Goncharuk > Priority: Critical > Fix For: 2.8 > > Attachments: PersistenceNodeRestartAfterCacheDropSelfTest.java, > ignite-gridparitition-nullpointer.zip > > > 1. Start 2 nodes with PDS > 2. Activate cluster > 3. Connect sqlline. > 4. Create table {{create table t1(a int, b varchar, primary key(a)) with > "ATOMICITY=TRANSACTIONAL_SNAPSHOT,backups=1";}} > 5. Stop node 1 > 6. Drop table {{drop table t1;}} > 7. Start node 1 > 8. Node 2 stopped by handler: > {noformat} > c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin>bin\ignite.bat server.xml -v -J-DID=1 > Ignite Command Line Startup, ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV > 2018 Copyright(C) Apache Software Foundation > [18:04:22,745][INFO][main][IgniteKernal] > >>> __________ ________________ > >>> / _/ ___/ |/ / _/_ __/ __/ > >>> _/ // (7 7 // / / / / _/ > >>> /___/\___/_/|_/___/ /_/ /___/ > >>> > >>> ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV > >>> 2018 Copyright(C) Apache Software Foundation > >>> > >>> Ignite documentation: http://ignite.apache.org > [18:04:22,745][INFO][main][IgniteKernal] Config URL: > file:/c:/Work/apache-ignite-2.7.0-SNAPSHOT-bin/server.xml > [18:04:22,760][INFO][main][IgniteKernal] IgniteConfiguration > [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, cal > lbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, > igfsPoolSize=8, dataStreamerPoolSize=8, utilityCacheP > oolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, > igniteHome=c:\Work\apache-ignite-2.7.0-SNAPSHO > T-bin, igniteWorkDir=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\work, > mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94 > fa3e, nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, marsh=BinaryMarshaller [], > marshLocJobs=false, daemon=false, p2pEnabl > ed=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, > metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpT > ime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, > sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, > reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, > clientReconnectDisabled=false, internalLsnr=null], segPlc=ST > OP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, > segChkFreq=10000, commSpi=TcpCommunicationSp > i [connectGate=null, > connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@22ff4249, > enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, > locHost=null, locPort=47100, locPortRange=1 > 00, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, > connTimeout=5000, maxConnTimeout=600000, r > econCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, > slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, us > ePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, > filterReachableAddresses=false, ackSndThreshold=32, una > ckedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, > boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRs > lvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@2d1ef81a[Count = > 1], stopping=false], evtSpi=org.apache.ignit > e.spi.eventstorage.NoopEventStorageSpi@4c402120, colSpi=NoopCollisionSpi [], > deploySpi=LocalDeploymentSpi [], indexingSp > i=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@815b41f, > addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encry > ption.noop.NoopEncryptionSpi@5542c4ed, clientMode=false, > rebalanceThreadPoolSize=1, txCfg=TransactionConfiguration [txSe > rEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, > dfltTxTimeout=0, txTimeoutOnPartitionMapExch > ange=0, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, > tmLookupClsName=null, txManagerFactory=null, useJtaSync=fa > lse], cacheSanityCheckEnabled=true, discoStartupDelay=60000, > deployMode=SHARED, p2pMissedCacheSize=100, locHost=127.0.0. > 1, timeSrvPortBase=31100, timeSrvPortRange=100, > failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailu > reDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, > connectorCfg=ConnectorConfiguration [jettyPath=null, hos > t=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, > rcvBufSize=32768, idleQryCurTimeout=600000, idleQry > CurCheckFreq=60000, sndQueueLimit=0, selectorCnt=4, idleTimeout=7000, > sslEnabled=false, sslClientAuth=false, sslCtxFacto > ry=null, sslFactory=null, portRange=100, threadPoolSize=8, > msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCf > g=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, > aff=null, grpName=null], classLdr=null, ss > lCtxFactory=null, platformCfg=null, binaryCfg=BinaryConfiguration > [idMapper=null, nameMapper=null, serializer=null, comp > actFooter=true], memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration > [sysRegionInitSize=52428800, sysRegionMaxSize= > 104857600, pageSize=0, concLvl=0, dfltDataRegConf=DataRegionConfiguration > [name=default, maxSize=104857600, initSize=524 > 28800, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, > emptyPagesPoolSize=100, metricsEnabled=false, me > tricsSubIntervalCount=5, metricsRateTimeInterval=60000, > persistenceEnabled=true, checkpointPageBufSize=0], dataRegions=D > ataRegionConfiguration[] [DataRegionConfiguration [name=dataRegion1, > maxSize=209715200, initSize=104857600, swapPath=nul > l, pageEvictionMode=RANDOM_LRU, evictionThreshold=0.9, > emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalC > ount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, > checkpointPageBufSize=0], DataRegionConfiguration [name= > dataRegion2, maxSize=104857600, initSize=52428800, swapPath=null, > pageEvictionMode=RANDOM_LRU, evictionThreshold=0.9, em > ptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, > metricsRateTimeInterval=60000, persistenceEnabled > =true, checkpointPageBufSize=0]], storagePath=null, checkpointFreq=1000, > lockWaitTime=10000, checkpointThreads=4, checkp > ointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, > walSegments=10, walSegmentSize=67108864, walPat > h=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, > walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFl > ushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, > alwaysWriteFullPages=false, fileIOFactory=org.apache.i > gnite.internal.processors.cache.persistence.file.AsyncFileIOFactory@3cd3e762, > metricsSubIntervalCnt=5, metricsRateTimeIn > terval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, > walCompactionEnabled=false, walCompactionL > evel=1, checkpointReadLockTimeout=null], activeOnStart=true, > autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=nu > ll, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, > portRange=100, sockSndBufSize=0, sockRcvBufSize=0, t > cpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, > jdbcEnabled=true, odbcEnabled=true, thinCliE > nabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, > sslClientAuth=false, sslCtxFactory=null], mvccVacuumThreadCn > t=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, > commFailureRslvr=null] > [18:04:22,760][INFO][main][IgniteKernal] Daemon mode: off > [18:04:22,760][INFO][main][IgniteKernal] OS: Windows 10 10.0 amd64 > [18:04:22,760][INFO][main][IgniteKernal] OS user: gridgain > [18:04:22,760][INFO][main][IgniteKernal] PID: 836 > [18:04:22,760][INFO][main][IgniteKernal] Language runtime: Java Platform API > Specification ver. 1.8 > [18:04:22,760][INFO][main][IgniteKernal] VM information: Java(TM) SE Runtime > Environment 1.8.0_181-b13 Oracle Corporatio > n Java HotSpot(TM) 64-Bit Server VM 25.181-b13 > [18:04:22,776][INFO][main][IgniteKernal] VM total memory: 0.96GB > [18:04:22,776][INFO][main][IgniteKernal] Remote Management [restart: on, > REST: on, JMX (remote: on, port: 49148, auth: o > ff, ssl: off)] > [18:04:22,776][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=false, > config=null] > [18:04:22,776][INFO][main][IgniteKernal] > IGNITE_HOME=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin > [18:04:22,776][INFO][main][IgniteKernal] VM arguments: [-XX:+AggressiveOpts, > -Xms1g, -Xmx1g, -XX:MaxMetaspaceSize=256m, > -DIGNITE_QUIET=false, > -DIGNITE_SUCCESS_FILE=c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\work\ignite_success_37805c22-8b1e-4 > 14e-86e2-e3af2e5e67df, -Dcom.sun.management.jmxremote, > -Dcom.sun.management.jmxremote.port=49148, -Dcom.sun.management.j > mxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, > -DIGNITE_HOME=c:\Work\apache-ignite-2.7.0-SNAPSHO > T-bin, -DIGNITE_PROG_NAME=ignite.bat, -DID=1] > [18:04:22,776][INFO][main][IgniteKernal] System cache's DataRegion size is > configured to 50 MB. Use DataStorageConfigura > tion.systemRegionInitialSize property to change the setting. > [18:04:22,776][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' > dataRegion: ['ignite-sys-cache']] > [18:04:22,776][WARNING][main][IgniteKernal] Please set system property > '-Djava.net.preferIPv4Stack=true' to avoid possib > le problems in mixed environments. > [18:04:22,776][WARNING][pub-#19][GridDiagnostic] This operating system has > been tested less rigorously: Windows 10 10.0 > amd64. Our team will appreciate the feedback if you experience any problems > running ignite in this environment. > [18:04:22,776][INFO][main][IgniteKernal] 3-rd party licenses can be found at: > c:\Work\apache-ignite-2.7.0-SNAPSHOT-bin\l > ibs\licenses > [18:04:22,829][INFO][main][IgnitePluginProcessor] Configured plugins: > [18:04:22,829][INFO][main][IgnitePluginProcessor] ^-- None > [18:04:22,829][INFO][main][IgnitePluginProcessor] > [18:04:22,845][INFO][main][FailureProcessor] Configured failure handler: > [hnd=StopNodeOrHaltFailureHandler [tryStop=fals > e, timeout=0, super=AbstractFailureHandler > [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED]]]] > [18:04:22,883][INFO][main][TcpCommunicationSpi] Successfully bound > communication NIO server to TCP port [port=47100, loc > Host=/127.0.0.1, selectorsCnt=4, selectorSpins=0, pairedConn=false] > [18:04:22,884][WARNING][main][TcpCommunicationSpi] Message queue limit is set > to 0 which may lead to potential OOMEs whe > n running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message > queues growth on sender and receiver sides > . > [18:04:22,901][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to > enable configure any GridCheckpointSpi imp > lementation) > [18:04:22,919][WARNING][main][GridCollisionManager] Collision resolution is > disabled (all jobs will be activated upon ar > rival). > [18:04:22,926][INFO][main][IgniteKernal] Security status [authentication=off, > tls/ssl=off] > [18:04:23,003][INFO][main][CacheObjectBinaryProcessorImpl] Resolved directory > for serialized binary metadata: c:\Work\ap > ache-ignite-2.7.0-SNAPSHOT-bin\work\binary_meta\1 > [18:04:23,125][INFO][main][FilePageStoreManager] Resolved page store work > directory: c:\Work\apache-ignite-2.7.0-SNAPSHO > T-bin\work\db\1 > [18:04:23,127][INFO][main][FileWriteAheadLogManager] Resolved write ahead log > work directory: c:\Work\apache-ignite-2.7. > 0-SNAPSHOT-bin\work\db\wal\1 > [18:04:23,128][INFO][main][FileWriteAheadLogManager] Resolved write ahead log > archive directory: c:\Work\apache-ignite-2 > .7.0-SNAPSHOT-bin\work\db\wal\archive\1 > [18:04:23,415][INFO][main][FileWriteAheadLogManager] Started write-ahead log > manager [mode=LOG_ONLY] > [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] Page eviction > mode set for [dataRegion1] data will have no > effect because the oldest pages are evicted automatically if Ignite > persistence is enabled. > [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] Page eviction > mode set for [dataRegion2] data will have no > effect because the oldest pages are evicted automatically if Ignite > persistence is enabled. > [18:04:23,415][WARNING][main][GridCacheDatabaseSharedManager] > DataRegionConfiguration.maxWalArchiveSize instead DataRegi > onConfiguration.walHistorySize would be used for removing old archive wal > files > [18:04:23,462][INFO][main][PartitionsEvictManager] Evict partition permits=2 > [18:04:23,631][INFO][main][ClientListenerProcessor] Client connector > processor has started on TCP port 10800 > [18:04:23,663][INFO][main][GridTcpRestProtocol] Command protocol successfully > started [name=TCP binary, host=/127.0.0.1, > port=11211] > [18:04:23,763][INFO][main][IgniteKernal] Non-loopback local IPs: 172.25.4.52, > fe80:0:0:0:2d2b:5bfb:e052:9fb0%eth5 > [18:04:23,763][INFO][main][IgniteKernal] Enabled local MACs: 54A050520B61 > [18:04:23,763][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint > status [startMarker=null, endMarker=null] > [18:04:23,779][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, tableSize=1,9 Mi > B, checkpointBuffer=100,0 MiB] > [18:04:23,779][INFO][main][GridCacheDatabaseSharedManager] Checking memory > state [lastValidPos=FileWALPointer [idx=0, fi > leOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000 > 000] > [18:04:23,794][INFO][main][GridCacheDatabaseSharedManager] Applying lost > cache updates since last checkpoint record [las > tMarked=FileWALPointer [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [18:04:23,794][INFO][main][GridCacheDatabaseSharedManager] Finished applying > WAL changes [updatesApplied=0, time=0ms] > [18:04:23,816][INFO][main][TcpDiscoverySpi] Connection check threshold is > calculated: 10000 > [18:04:23,832][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port > [port=47500, localHost=127.0.0.1/127.0.0.1, l > ocNodeId=d02069db-6d0b-4a40-b185-1d95fa330853] > [18:04:32,790][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=61618] > [18:04:32,822][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > spawning a new thread for connection [rmtAddr=/12 > 7.0.0.1, rmtPort=61618] > [18:04:32,822][INFO][tcp-disco-sock-reader-#4][TcpDiscoverySpi] Started > serving remote node connection [rmtAddr=/127.0.0 > .1:61618, rmtPort=61618] > [18:04:32,839][INFO][tcp-disco-sock-reader-#4][TcpDiscoverySpi] Finished > serving remote node connection [rmtAddr=/127.0. > 0.1:61618, rmtPort=61618 > [18:04:43,227][INFO][disco-notifier-worker-#41][MvccProcessorImpl] Assigned > mvcc coordinator [crd=MvccCoordinator [nodeI > d=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0]], > crdNode=TcpDiscoveryNode [id=d02069db-6d0b-4a40-b185-1d95fa330853, > addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127 > .0.0.1:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1540479863832, loc=true, ver=2.7.0#19700101-sha1:00 > 000000, isClient=false]] > [18:04:43,310][INFO][main][IgniteKernal] Performance suggestions for grid > (fix if possible) > [18:04:43,310][INFO][main][IgniteKernal] To disable, set > -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > [18:04:43,310][INFO][main][IgniteKernal] ^-- Enable G1 Garbage Collector > (add '-XX:+UseG1GC' to JVM options) > [18:04:43,310][INFO][main][IgniteKernal] ^-- Set max direct memory size if > getting 'OOME: Direct buffer memory' (add ' > -XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) > [18:04:43,310][INFO][main][IgniteKernal] ^-- Disable processing of calls to > System.gc() (add '-XX:+DisableExplicitGC' > to JVM options) > [18:04:43,310][INFO][main][IgniteKernal] Refer to this page for more > performance suggestions: https://apacheignite.readm > e.io/docs/jvm-and-system-tuning > [18:04:43,310][INFO][main][IgniteKernal] > [18:04:43,310][INFO][main][IgniteKernal] To start Console Management & > Monitoring run ignitevisorcmd.{sh|bat} > [18:04:43,310][INFO][main][IgniteKernal] Data Regions Configured: > [18:04:43,310][INFO][main][IgniteKernal] ^-- default [initSize=50,0 MiB, > maxSize=100,0 MiB, persistence=true] > [18:04:43,310][INFO][main][IgniteKernal] ^-- dataRegion1 [initSize=100,0 > MiB, maxSize=200,0 MiB, persistence=true] > [18:04:43,310][INFO][main][IgniteKernal] ^-- dataRegion2 [initSize=50,0 > MiB, maxSize=100,0 MiB, persistence=true] > [18:04:43,310][INFO][main][IgniteKernal] > >>> +------------------------------------------+ > >>> Ignite ver. 2.7.0-SNAPSHOT#19700101-sha1:DEV > >>> +------------------------------------------+ > >>> OS name: Windows 10 10.0 amd64 > >>> CPU(s): 8 > >>> Heap: 1.0GB > >>> VM name: 836@work-pc > >>> Local node [ID=D02069DB-6D0B-4A40-B185-1D95FA330853, order=1, > >>> clientMode=false] > >>> Local node addresses: [127.0.0.1] > >>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500 > [18:04:43,310][INFO][main][IgniteKernal] >>> Ignite cluster is not active > (limited functionality available). Use control > .(sh|bat) script or IgniteCluster interface to activate. > [18:04:43,310][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1, > locNode=d02069db, servers=1, clients=0, state > =INACTIVE, CPUs=8, offheap=0.39GB, heap=1.0GB] > [18:04:43,341][INFO][rest-#45][GridClusterStateProcessor] Sending activate > request with BaselineTopology BaselineTopolog > y [id=0, branchingHash=49, branchingType='New BaselineTopology', > baselineNodes=[1]] > [18:04:43,357][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] > Received activate request with BaselineTopolo > gy[id=0] > [18:04:43,357][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] > Started state transition: true > [18:04:43,357][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=1, minorTo > pVer=1], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=true, crd=true, > evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4a > 40-b185-1d95fa330853, customEvt=ChangeGlobalStateMessage > [id=a6641cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqId=a2c > c8ce1-aff1-4600-aa51-a600474de991, > initiatingNodeId=d02069db-6d0b-4a40-b185-1d95fa330853, activate=true, > baselineTopolog > y=BaselineTopology [id=0, branchingHash=49, branchingType='New > BaselineTopology', baselineNodes=[1]], forceChangeBaselin > eTopology=false, timestamp=1540479883341], allowMerge=false] > [18:04:43,357][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Start activation process [nodeId=d02069db-6d0 > b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1]] > [18:04:43,357][INFO][exchange-worker-#43][FilePageStoreManager] Resolved page > store work directory: c:\Work\apache-ignit > e-2.7.0-SNAPSHOT-bin\work\db\1 > [18:04:43,357][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resolved > write ahead log work directory: c:\Work\apa > che-ignite-2.7.0-SNAPSHOT-bin\work\db\wal\1 > [18:04:43,372][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resolved > write ahead log archive directory: c:\Work\ > apache-ignite-2.7.0-SNAPSHOT-bin\work\db\wal\archive\1 > [18:04:43,372][INFO][exchange-worker-#43][FileWriteAheadLogManager] Started > write-ahead log manager [mode=LOG_ONLY] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=200,0 MiB, pages=49624, t > ableSize=3,9 MiB, checkpointBuffer=200,0 MiB] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, t > ableSize=1,9 MiB, checkpointBuffer=100,0 MiB] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, t > ableSize=1,9 MiB, checkpointBuffer=100,0 MiB] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, t > ableSize=1,9 MiB, checkpointBuffer=100,0 MiB] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, t > ableSize=1,9 MiB, checkpointBuffer=100,0 MiB] > [18:04:43,379][INFO][exchange-worker-#43][PageMemoryImpl] Started page memory > [memoryAllocated=100,0 MiB, pages=24808, t > ableSize=1,9 MiB, checkpointBuffer=100,0 MiB] > [18:04:43,395][INFO][exchange-worker-#43][msg] Components activation > performed in 38 ms. > [18:04:43,410][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Read checkpoint status [startMarker=null, endM > arker=null] > [18:04:43,410][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Checking memory state [lastValidPos=FileWALPoi > nter [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, > len=0], lastCheckpointId=00000000-0000-0000 > -0000-000000000000] > [18:04:43,410][INFO][exchange-worker-#43][FileWriteAheadLogManager] Resuming > logging to WAL segment [file=c:\Work\apache > -ignite-2.7.0-SNAPSHOT-bin\work\db\wal\1\0000000000000000.wal, offset=0, > ver=2] > [18:04:43,457][INFO][exchange-worker-#43][GridClusterStateProcessor] Writing > BaselineTopology[id=0] > [18:04:43,473][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Binary recovery performed in 78 ms. > [18:04:43,473][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Updating caches registry performed in 0 ms. > [18:04:43,611][INFO][exchange-worker-#43][GridCacheProcessor] Started cache > [name=ignite-sys-cache, id=-2100569601, memo > ryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, > backups=2147483647, mvcc=false], encryptionEnabled=fal > se] > [18:04:43,611][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches > starting performed in 138 ms. > [18:04:43,611][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Affinity initialization for started caches perform > ed in 0 ms. > [18:04:43,611][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Successfully activated caches [nodeId=d02069d > b-6d0b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1]] > [18:04:43,626][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:04:43,626][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Read checkpoint status [startMarker=null, endM > arker=null] > [18:04:43,626][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Applying lost cache updates since last checkpo > int record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [18:04:43,679][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Restoring partition state for local groups [cn > tPartStateWal=0, lastCheckpointId=00000000-0000-0000-0000-000000000000] > [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Finished restoring partition state for local g > roups [cntProcessed=100, cntPartStateWal=0, time=16ms] > [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Finished applying WAL changes [updatesApplied= > 0, time=69ms] > [18:04:43,695][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 69 ms. > [18:04:43,726][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Checkpointer initilialzation performed in 15 m > s. > [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityT > opologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1]] > [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Affinity changes (coordinator) applied in 0 m > s. > [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions validation performed in 0 ms. > [18:04:43,726][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions assignment performed in 0 ms. > [18:04:43,726][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Affinity recalculation (custom message) performed > in 0 ms. > [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Preparing Full Message performed in 16 ms. > [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Sending Full Message to all nodes performed i > n 0 ms. > [18:04:43,742][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1], err=null] > [18:04:43,742][INFO][disco-notifier-worker-#41][GridClusterStateProcessor] > Received state change finish message: true > [18:04:43,742][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Sending Full Message for AffinityTopologyVe > rsion [topVer=1, minorTopVer=1] performed in 0 ms. > [18:04:43,742][INFO][exchange-worker-#43][MvccProcessorImpl] Initialize local > node as mvcc coordinator [node=d02069db-6d > 0b-4a40-b185-1d95fa330853, crdVer=1540479883228] > [18:04:43,742][INFO][exchange-worker-#43][GridCacheProcessor] Finish proxy > initialization, cacheName=ignite-sys-cache, l > ocalNodeId=d02069db-6d0b-4a40-b185-1d95fa330853 > [18:04:43,758][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Completed partition exchange [localNode=d0206 > 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture > [topVer=AffinityTopologyVersion [topVer=1, min > orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode > [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis > t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, > intOrder=1, lastExchangeTime=1540479863832 > , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=1, min > orTopVer=1], durationFromInit=401] > [18:04:43,758][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=1, minorT > opVer=1], crd=true] > [18:04:43,758][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [t > op=AffinityTopologyVersion [topVer=1, minorTopVer=1], force=false, > evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18 > 5-1d95fa330853] > [18:04:43,758][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Checkpoint started [checkpointId=143137c0 > -86bf-44a0-9f9c-efba0161e74d, startPtr=FileWALPointer [idx=0, fileOff=119489, > len=1947], checkpointLockWait=0ms, checkpo > intLockHoldTime=15ms, walCpRecordFsyncDuration=16ms, pages=17, > reason='timeout'] > [18:04:43,773][INFO][sys-#54][GridClusterStateProcessor] Successfully > performed final activation steps [nodeId=d02069db- > 6d0b-4a40-b185-1d95fa330853, client=false, topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1]] > [18:04:43,795][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Checkpoint finished [cpId=143137c0-86bf-4 > 4a0-9f9c-efba0161e74d, pages=17, markPos=FileWALPointer [idx=0, > fileOff=119489, len=1947], walSegmentsCleared=0, walSegm > entsCovered=[], markDuration=62ms, pagesWrite=0ms, fsync=22ms, total=84ms] > [18:04:44,729][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=0ms, reason='timeout'] > [18:04:48,037][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=61665] > [18:04:48,037][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > spawning a new thread for connection [rmtAddr=/12 > 7.0.0.1, rmtPort=61665] > [18:04:48,041][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Started > serving remote node connection [rmtAddr=/127.0.0 > .1:61665, rmtPort=61665] > [18:04:48,063][INFO][tcp-disco-sock-reader-#5][TcpDiscoverySpi] Finished > serving remote node connection [rmtAddr=/127.0. > 0.1:61665, rmtPort=61665 > [18:04:48,069][INFO][tcp-disco-msg-worker-#2][GridEncryptionManager] Joining > node doesn't have encryption data [node=5df > 37010-aaf1-4c1c-88fb-53b8b178c4f5] > [18:04:48,093][INFO][tcp-disco-msg-worker-#2][TcpDiscoverySpi] New next node > [newNext=TcpDiscoveryNode [id=5df37010-aaf1 > -4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet > [/127.0.0.1:47501], discPort=47501, order=0, int > Order=2, lastExchangeTime=1540479888049, loc=false, > ver=2.7.0#19700101-sha1:00000000, isClient=false]] > [18:04:48,129][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=61667] > [18:04:48,129][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > spawning a new thread for connection [rmtAddr=/12 > 7.0.0.1, rmtPort=61667] > [18:04:48,130][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Started > serving remote node connection [rmtAddr=/127.0.0 > .1:61667, rmtPort=61667] > [18:04:48,136][INFO][disco-event-worker-#42][GridDiscoveryManager] Added new > node to topology: TcpDiscoveryNode [id=5df3 > 7010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], > sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, ord > er=2, intOrder=2, lastExchangeTime=1540479888049, loc=false, > ver=2.7.0#19700101-sha1:00000000, isClient=false] > [18:04:48,137][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology > snapshot [ver=2, locNode=d02069db, servers=2 > , clients=0, state=ACTIVE, CPUs=8, offheap=0.78GB, heap=2.0GB] > [18:04:48,137][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- > Baseline [id=0, size=1, online=1, offline=0] > [18:04:48,138][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=2, minorTo > pVer=0], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, > evt=NODE_JOINED, evtNode=5df37010-aaf1-4c1c-88fb- > 53b8b178c4f5, customEvt=null, allowMerge=true] > [18:04:48,145][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=DISTRIBUTED] > [18:04:48,148][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partitions release latch > : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch > [id=exchange, topVer=AffinityTopologyVersion [t > opVer=2, minorTopVer=0]]] > [18:04:48,148][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:04:48,148][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 0 ms. > [18:04:48,149][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=2, minorT > opVer=0], crd=true] > [18:04:48,725][INFO][grid-nio-worker-tcp-comm-0-#24][TcpCommunicationSpi] > Accepted incoming communication connection [lo > cAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:61669] > [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Coordinator > received single message [ver=AffinityTopology > Version [topVer=2, minorTopVer=0], node=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, > allReceived=true] > [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Coordinator > received all messages, try merge [ver=Affinit > yTopologyVersion [topVer=2, minorTopVer=0]] > [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Exchanges > merging performed in 0 ms. > [18:04:48,741][INFO][sys-#56][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityTopologyVersi > on [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, > minorTopVer=0]] > [18:04:48,741][INFO][sys-#56][CacheAffinitySharedManager] Affinity > recalculation (on server join) performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Affinity > changes (coordinator) applied in 15 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Partitions > validation performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Partitions > assignment performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Preparing Full > Message performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Sending Full > Message performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Sending Full > Message to all nodes performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Finish > exchange future [startVer=AffinityTopologyVersion > [topVer=2, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=2, > minorTopVer=0], err=null] > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Detecting lost > partitions performed in 0 ms. > [18:04:48,756][INFO][sys-#56][GridDhtPartitionsExchangeFuture] Completed > partition exchange [localNode=d02069db-6d0b-4a4 > 0-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], > evt=NODE_JOINED, evtNode=TcpDiscoveryNode > [id=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAd > drs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, > lastExchangeTime=1540479888049, loc=false, ver=2.7. > 0#19700101-sha1:00000000, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], duratio > nFromInit=620] > [18:04:48,756][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [t > op=AffinityTopologyVersion [topVer=2, minorTopVer=0], force=false, > evt=NODE_JOINED, node=5df37010-aaf1-4c1c-88fb-53b8b17 > 8c4f5] > [18:04:49,769][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=3ms, reason='timeout'] > [18:04:55,817][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=1ms, reason='timeout'] > [18:05:02,884][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=2ms, reason='timeout'] > [18:05:09,804][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=2, minorTo > pVer=1], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, > evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4 > a40-b185-1d95fa330853, customEvt=DynamicCacheChangeBatch > [id=c6741cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqs=Arra > yList [DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_T1, hasCfg=true, > nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, cli > entStartOnly=false, stop=false, destroy=false, disabledAfterStartfalse]], > exchangeActions=ExchangeActions [startCaches=[ > SQL_PUBLIC_T1], stopCaches=null, startGrps=[SQL_PUBLIC_T1], stopGrps=[], > resetParts=null, stateChangeRequest=null], star > tCaches=false], allowMerge=false] > [18:05:09,804][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Updating caches registry performed in 0 ms. > [18:05:09,875][INFO][exchange-worker-#43][MvccProcessorImpl] Mvcc processor > started. > [18:05:09,912][INFO][exchange-worker-#43][GridCacheProcessor] Started cache > [name=SQL_PUBLIC_T1, id=-1691533662, memoryP > olicyName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL_SNAPSHOT, > backups=1, mvcc=true], encryptionEnabled=false] > [18:05:09,912][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches > starting performed in 108 ms. > [18:05:09,915][INFO][exchange-worker-#43][GridAffinityAssignmentCache] Local > node affinity assignment distribution is no > t ideal [cache=SQL_PUBLIC_T1, expectedPrimary=1024,00, actualPrimary=1024, > expectedBackups=1024,00, actualBackups=0, war > ningThreshold=50,00%] > [18:05:09,916][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Affinity initialization for started caches perform > ed in 3 ms. > [18:05:09,917][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=DISTRIBUTED] > [18:05:09,934][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Checkpoint started [checkpointId=39c009dc > -7bd4-465e-8a4d-156ea670668b, startPtr=FileWALPointer [idx=0, fileOff=131199, > len=1953], checkpointLockWait=5ms, checkpo > intLockHoldTime=1ms, walCpRecordFsyncDuration=9ms, pages=13, reason='timeout'] > [18:05:09,945][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partitions release latch > : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch > [id=exchange, topVer=AffinityTopologyVersion [t > opVer=2, minorTopVer=1]]] > [18:05:09,946][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:05:09,947][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Checkpoint finished [cpId=39c009dc-7bd4-4 > 65e-8a4d-156ea670668b, pages=13, markPos=FileWALPointer [idx=0, > fileOff=131199, len=1953], walSegmentsCleared=0, walSegm > entsCovered=[], markDuration=22ms, pagesWrite=2ms, fsync=11ms, total=40ms] > [18:05:09,963][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 17 ms. > [18:05:10,014][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Coordinator received single message [ver=Affi > nityTopologyVersion [topVer=2, minorTopVer=1], > node=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, allReceived=true] > [18:05:10,014][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityT > opologyVersion [topVer=2, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=2, minorTopVer=1]] > [18:05:10,015][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Affinity changes (coordinator) applied in 0 m > s. > [18:05:10,016][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions validation performed in 1 ms. > [18:05:10,020][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions assignment performed in 4 ms. > [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Preparing Full Message performed in 4 ms. > [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Sending Full Message performed in 0 ms. > [18:05:10,025][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Sending Full Message to all nodes performed i > n 0 ms. > [18:05:10,026][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopo > logyVersion [topVer=2, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=2, minorTopVer=1], err=null] > [18:05:10,029][INFO][exchange-worker-#43][GridCacheProcessor] Finish proxy > initialization, cacheName=SQL_PUBLIC_T1, loca > lNodeId=d02069db-6d0b-4a40-b185-1d95fa330853 > [18:05:10,030][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Completed partition exchange [localNode=d0206 > 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture > [topVer=AffinityTopologyVersion [topVer=2, min > orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode > [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis > t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, > intOrder=1, lastExchangeTime=1540479909804 > , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=2, min > orTopVer=1], durationFromInit=222] > [18:05:10,030][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=2, minorT > opVer=1], crd=true] > [18:05:10,031][INFO][pub-#70][GridCacheDatabaseSharedManager] Finished > indexes rebuilding for cache [name=SQL_PUBLIC_T1, > grpName=null] > [18:05:10,033][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [t > op=AffinityTopologyVersion [topVer=2, minorTopVer=1], force=false, > evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18 > 5-1d95fa330853] > [18:05:10,919][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=4ms, reason='timeout'] > [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] Node left > topology: TcpDiscoveryNode [id=5df37010-aaf > 1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet > [/127.0.0.1:47501], discPort=47501, order=2, in > tOrder=2, lastExchangeTime=1540479888049, loc=false, > ver=2.7.0#19700101-sha1:00000000, isClient=false] > [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology > snapshot [ver=3, locNode=d02069db, servers=1 > , clients=0, state=ACTIVE, CPUs=8, offheap=0.39GB, heap=1.0GB] > [18:05:14,456][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- > Baseline [id=0, size=1, online=1, offline=0] > [18:05:14,488][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=3, minorTo > pVer=0], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, > evt=NODE_LEFT, evtNode=5df37010-aaf1-4c1c-88fb-53 > b8b178c4f5, customEvt=null, allowMerge=true] > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=DISTRIBUTED] > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partitions release latch > : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch > [id=exchange, topVer=AffinityTopologyVersion [t > opVer=3, minorTopVer=0]]] > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:05:14,488][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 0 ms. > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Coordinator received all messages, try merge > [ver=AffinityTopologyVersion [topVer=3, minorTopVer=0]] > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Exchanges merging performed in 0 ms. > [18:05:14,488][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityT > opologyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion > [topVer=3, minorTopVer=0]] > [18:05:14,488][INFO][exchange-worker-#43][GridAffinityAssignmentCache] Local > node affinity assignment distribution is no > t ideal [cache=SQL_PUBLIC_T1, expectedPrimary=1024,00, actualPrimary=1024, > expectedBackups=1024,00, actualBackups=0, war > ningThreshold=50,00%] > [18:05:14,511][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Affinity recalculation (on server left) performed > in 23 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Affinity changes (coordinator) applied in 25 > ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions validation performed in 0 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Detecting lost partitions performed in 0 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Preparing Full Message performed in 0 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Sending Full Message to all nodes performed i > n 0 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopo > logyVersion [topVer=3, minorTopVer=0], resVer=AffinityTopologyVersion > [topVer=3, minorTopVer=0], err=null] > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Detecting lost partitions performed in 0 ms. > [18:05:14,513][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Completed partition exchange [localNode=d0206 > 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture > [topVer=AffinityTopologyVersion [topVer=3, min > orTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode > [id=5df37010-aaf1-4c1c-88fb-53b8b178c4f5, addrs=ArrayList [127.0.0. > 1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, > intOrder=2, lastExchangeTime=1540479888049, loc=false > , ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0 > ], durationFromInit=15] > [18:05:14,513][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=3, minorT > opVer=0], crd=true] > [18:05:14,513][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [t > op=AffinityTopologyVersion [topVer=3, minorTopVer=0], force=false, > evt=NODE_LEFT, node=5df37010-aaf1-4c1c-88fb-53b8b178c > 4f5] > [18:05:14,982][INFO][tcp-disco-sock-reader-#6][TcpDiscoverySpi] Finished > serving remote node connection [rmtAddr=/127.0. > 0.1:61667, rmtPort=61667 > [18:05:21,967][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=5ms, reason='timeout'] > [18:05:23,818][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=3, minorTo > pVer=1], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, > evt=DISCOVERY_CUSTOM_EVT, evtNode=d02069db-6d0b-4 > a40-b185-1d95fa330853, customEvt=DynamicCacheChangeBatch > [id=ea741cba661-606014c4-c468-4812-86de-7e8ae4cc30b9, reqs=Arra > yList [DynamicCacheChangeRequest [cacheName=SQL_PUBLIC_T1, hasCfg=false, > nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, cl > ientStartOnly=false, stop=true, destroy=false, disabledAfterStartfalse]], > exchangeActions=ExchangeActions [startCaches=n > ull, stopCaches=[SQL_PUBLIC_T1], startGrps=[], stopGrps=[SQL_PUBLIC_T1, > destroy=true], resetParts=null, stateChangeReque > st=null], startCaches=false], allowMerge=false] > [18:05:23,819][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Updating caches registry performed in 1 ms. > [18:05:23,820][INFO][exchange-worker-#43][CacheAffinitySharedManager] Caches > starting performed in 0 ms. > [18:05:23,820][INFO][exchange-worker-#43][CacheAffinitySharedManager] > Affinity initialization for started caches perform > ed in 0 ms. > [18:05:23,820][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=DISTRIBUTED] > [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partitions release latch > : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch > [id=exchange, topVer=AffinityTopologyVersion [t > opVer=3, minorTopVer=1]]] > [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:05:23,821][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 0 ms. > [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityT > opologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=3, minorTopVer=1]] > [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Affinity changes (coordinator) applied in 0 m > s. > [18:05:23,821][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions validation performed in 0 ms. > [18:05:23,822][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Partitions assignment performed in 0 ms. > [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Preparing Full Message performed in 3 ms. > [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Sending Full Message to all nodes performed i > n 0 ms. > [18:05:23,825][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopo > logyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion > [topVer=3, minorTopVer=1], err=null] > [18:05:23,827][INFO][db-checkpoint-thread-#49][GridCacheDatabaseSharedManager] > Skipping checkpoint (no pages were modifi > ed) [checkpointLockWait=0ms, checkpointLockHoldTime=2ms, reason='caches stop'] > [18:05:23,833][INFO][exchange-worker-#43][GridCacheProcessor] Stopped cache > [cacheName=SQL_PUBLIC_T1] > [18:05:23,851][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Completed partition exchange [localNode=d0206 > 9db-6d0b-4a40-b185-1d95fa330853, exchange=GridDhtPartitionsExchangeFuture > [topVer=AffinityTopologyVersion [topVer=3, min > orTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode > [id=d02069db-6d0b-4a40-b185-1d95fa330853, addrs=ArrayLis > t [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, > intOrder=1, lastExchangeTime=1540479914456 > , loc=true, ver=2.7.0#19700101-sha1:00000000, isClient=false], done=true], > topVer=AffinityTopologyVersion [topVer=3, min > orTopVer=1], durationFromInit=33] > [18:05:23,851][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=3, minorT > opVer=1], crd=true] > [18:05:23,853][INFO][exchange-worker-#43][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [t > op=AffinityTopologyVersion [topVer=3, minorTopVer=1], force=false, > evt=DISCOVERY_CUSTOM_EVT, node=d02069db-6d0b-4a40-b18 > 5-1d95fa330853] > [18:05:36,638][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=61721] > [18:05:36,638][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > spawning a new thread for connection [rmtAddr=/12 > 7.0.0.1, rmtPort=61721] > [18:05:36,638][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Started > serving remote node connection [rmtAddr=/127.0.0 > .1:61721, rmtPort=61721] > [18:05:36,669][INFO][tcp-disco-sock-reader-#7][TcpDiscoverySpi] Finished > serving remote node connection [rmtAddr=/127.0. > 0.1:61721, rmtPort=61721 > [18:05:36,685][INFO][tcp-disco-msg-worker-#2][GridEncryptionManager] Joining > node doesn't have encryption data [node=d07 > 9c11d-bd7d-4d65-93ab-dee8d0b280e3] > [18:05:36,685][INFO][tcp-disco-msg-worker-#2][TcpDiscoverySpi] New next node > [newNext=TcpDiscoveryNode [id=d079c11d-bd7d > -4d65-93ab-dee8d0b280e3, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet > [/127.0.0.1:47501], discPort=47501, order=0, int > Order=3, lastExchangeTime=1540479936669, loc=false, > ver=2.7.0#19700101-sha1:00000000, isClient=false]] > [18:05:36,738][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > accepted incoming connection [rmtAddr=/127.0.0.1, > rmtPort=61723] > [18:05:36,738][INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery > spawning a new thread for connection [rmtAddr=/12 > 7.0.0.1, rmtPort=61723] > [18:05:36,738][INFO][tcp-disco-sock-reader-#8][TcpDiscoverySpi] Started > serving remote node connection [rmtAddr=/127.0.0 > .1:61723, rmtPort=61723] > [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] Added new > node to topology: TcpDiscoveryNode [id=d079 > c11d-bd7d-4d65-93ab-dee8d0b280e3, addrs=ArrayList [127.0.0.1], > sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, ord > er=4, intOrder=3, lastExchangeTime=1540479936669, loc=false, > ver=2.7.0#19700101-sha1:00000000, isClient=false] > [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] Topology > snapshot [ver=4, locNode=d02069db, servers=2 > , clients=0, state=ACTIVE, CPUs=8, offheap=0.78GB, heap=2.0GB] > [18:05:36,754][INFO][disco-event-worker-#42][GridDiscoveryManager] ^-- > Baseline [id=0, size=1, online=1, offline=0] > [18:05:36,754][INFO][exchange-worker-#43][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=4, minorTo > pVer=0], mvccCrd=MvccCoordinator > [nodeId=d02069db-6d0b-4a40-b185-1d95fa330853, crdVer=1540479883228, > topVer=AffinityTopo > logyVersion [topVer=1, minorTopVer=0]], mvccCrdChange=false, crd=true, > evt=NODE_JOINED, evtNode=d079c11d-bd7d-4d65-93ab- > dee8d0b280e3, customEvt=null, allowMerge=true] > [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=DISTRIBUTED] > [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partitions release latch > : ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch > [id=exchange, topVer=AffinityTopologyVersion [t > opVer=4, minorTopVer=0]]] > [18:05:36,754][INFO][exchange-worker-#43][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [18:05:36,754][INFO][exchange-worker-#43][GridCacheDatabaseSharedManager] > Logical recovery performed in 0 ms. > [18:05:36,754][INFO][exchange-worker-#43][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=4, minorT > opVer=0], crd=true] > [18:05:37,423][INFO][grid-nio-worker-tcp-comm-1-#25][TcpCommunicationSpi] > Accepted incoming communication connection [lo > cAddr=/127.0.0.1:47100, rmtAddr=/127.0.0.1:61724] > [18:05:37,439][INFO][sys-#62][GridDhtPartitionsExchangeFuture] Coordinator > received single message [ver=AffinityTopology > Version [topVer=4, minorTopVer=0], node=d079c11d-bd7d-4d65-93ab-dee8d0b280e3, > allReceived=true] > [18:05:37,439][SEVERE][sys-#62][GridDhtPartitionsExchangeFuture] Failed to > notify listener: o.a.i.i.processors.cache.dis > tributed.dht.preloader.GridDhtPartitionsExchangeFuture$2@3e21b701 > java.lang.NullPointerException > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti > onExchangeManager.java:835) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP > artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process > SingleMessage(GridDhtPartitionsExchangeFuture.java:2790) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$ > 100(GridDhtPartitionsExchangeFuture.java:139) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2612) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei > veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr > idCachePartitionExchangeManager.java:1656) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE > xchangeManager.java:142) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:375) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:333) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3102) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3081) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119 > 7) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [18:05:37,455][SEVERE][sys-#62][GridCacheIoManager] Failed processing message > [senderId=d079c11d-bd7d-4d65-93ab-dee8d0b2 > 80e3, msg=GridDhtPartitionsSingleMessage [parts=HashMap > {-2100569601=GridDhtPartitionMap [moving=0, top=AffinityTopology > Version [topVer=-1, minorTopVer=0], updateSeq=2, size=0], > -1691533662=GridDhtPartitionMap [moving=0, top=AffinityTopolog > yVersion [topVer=4, minorTopVer=0], updateSeq=2, size=0]}, partCntrs=HashMap > {-2100569601=CachePartitionPartialCountersM > ap {}, -1691533662=CachePartitionPartialCountersMap {}}, partsSizes=null, > partHistCntrs=null, err=null, client=false, co > mpress=false, finishMsg=null, activeQryTrackers=GridLongList [idx=0, arr=[]], > super=GridDhtPartitionsAbstractMessage [ex > chId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=4, > minorTopVer=0], discoEvt=null, nodeId=d079c11 > d, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1540479929882, > nodeOrder=0], super=GridCacheMessage [msgI > d=1, depInfo=null, err=null, skipPrepare=false]]]] > java.lang.NullPointerException > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti > onExchangeManager.java:835) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP > artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process > SingleMessage(GridDhtPartitionsExchangeFuture.java:2790) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$ > 100(GridDhtPartitionsExchangeFuture.java:139) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2612) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei > veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr > idCachePartitionExchangeManager.java:1656) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE > xchangeManager.java:142) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:375) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:333) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3102) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3081) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119 > 7) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [18:05:37,455][SEVERE][sys-#62][] Critical system error detected. Will be > handled accordingly to configured handler [hnd > =StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableS > et [SYSTEM_WORKER_BLOCKED]]], failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.NullPointerException]] > java.lang.NullPointerException > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartiti > onExchangeManager.java:835) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updateP > artitionSingleMap(GridDhtPartitionsExchangeFuture.java:4031) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.process > SingleMessage(GridDhtPartitionsExchangeFuture.java:2790) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$ > 100(GridDhtPartitionsExchangeFuture.java:139) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2612) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply > (GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:385) > at > org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355) > at > org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onRecei > veSingleMessage(GridDhtPartitionsExchangeFuture.java:2600) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(Gr > idCachePartitionExchangeManager.java:1656) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionE > xchangeManager.java:142) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:375) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionE > xchangeManager.java:333) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3102) > at > org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCacheP > artitionExchangeManager.java:3081) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > at > org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:119 > 7) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > at > org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > [18:05:37,455][WARNING][sys-#62][FailureProcessor] No deadlocked threads > detected. > [18:05:37,486][WARNING][sys-#62][FailureProcessor] Thread dump at 2018/10/25 > 18:05:37 MSK > Thread [name="tcp-disco-sock-reader-#8", id=112, state=RUNNABLE, blockCnt=0, > waitCnt=0] > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > - locked java.io.BufferedInputStream@69777ba6 > at > o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) > at > java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2663) > at > java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2679) > at > java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3156) > at > java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:862) > at java.io.ObjectInputStream.<init>(ObjectInputStream.java:358) > at > o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:43) > at > o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:137) > at > o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94) > at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10085) > at > o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6246) > at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > Thread [name="pool-3-thread-1", id=110, state=TIMED_WAITING, blockCnt=0, > waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@36a3688e, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="client-connector-#73", id=109, state=WAITING, blockCnt=0, > waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="pub-#72", id=108, state=TIMED_WAITING, blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="pub-#70", id=106, state=TIMED_WAITING, blockCnt=0, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#69", id=105, state=TIMED_WAITING, blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#68", id=104, state=TIMED_WAITING, blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="vacuum-cleaner-#66", id=102, state=WAITING, blockCnt=0, > waitCnt=3] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cc7d605, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2184) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="vacuum-cleaner-#65", id=101, state=WAITING, blockCnt=2, > waitCnt=10] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cc7d605, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumWorker.body(MvccProcessorImpl.java:2184) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="vacuum-scheduler-#64", id=100, state=TIMED_WAITING, blockCnt=0, > waitCnt=8] > at java.lang.Thread.sleep(Native Method) > at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774) > at > o.a.i.i.processors.cache.mvcc.MvccProcessorImpl$VacuumScheduler.body(MvccProcessorImpl.java:2158) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="query-#63", id=99, state=TIMED_WAITING, blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@479516d3, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#62", id=98, state=RUNNABLE, blockCnt=0, waitCnt=1] > at sun.management.ThreadImpl.dumpThreads0(Native Method) > at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) > at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1364) > at > o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:132) > - locked o.a.i.i.processors.failure.FailureProcessor@5a6c304a > at > o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:108) > at > o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1071) > at > o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581) > at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380) > at > o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306) > at > o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101) > at > o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295) > at > o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569) > at > o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197) > at > o.a.i.i.managers.communication.GridIoManager.access$4200(GridIoManager.java:127) > at > o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1093) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@69455d2e > Thread [name="client-connector-#61", id=97, state=WAITING, blockCnt=0, > waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#60", id=96, state=TIMED_WAITING, blockCnt=0, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#59", id=95, state=TIMED_WAITING, blockCnt=0, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="jdbc-request-handler-worker-#58", id=94, state=WAITING, > blockCnt=0, waitCnt=3] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@52c43e80, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > o.a.i.i.processors.odbc.jdbc.JdbcRequestHandlerWorker.body(JdbcRequestHandlerWorker.java:85) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="client-connector-#57", id=93, state=WAITING, blockCnt=0, > waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54dbe05e, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#56", id=90, state=TIMED_WAITING, blockCnt=0, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="srvc-deploy-#55", id=87, state=WAITING, blockCnt=0, waitCnt=6] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20c1ad23, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#54", id=86, state=TIMED_WAITING, blockCnt=3, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="checkpoint-runner-#53", id=85, state=WAITING, blockCnt=4, > waitCnt=781] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="checkpoint-runner-#52", id=84, state=WAITING, blockCnt=6, > waitCnt=797] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="checkpoint-runner-#51", id=83, state=WAITING, blockCnt=3, > waitCnt=823] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="checkpoint-runner-#50", id=82, state=WAITING, blockCnt=4, > waitCnt=804] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4f8d1466, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="db-checkpoint-thread-#49", id=81, state=TIMED_WAITING, > blockCnt=4, waitCnt=225] > Lock > [object=o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer@745883bd, > ownerName=nu > ll, ownerId=-1] > at java.lang.Object.wait(Native Method) > at > o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.waitCheckpointEvent(GridCach > eDatabaseSharedManager.java:3537) > at > o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseShared > Manager.java:3085) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-#48", id=80, state=TIMED_WAITING, blockCnt=0, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@618d35a2, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="wal-segment-syncer-#47", id=79, state=TIMED_WAITING, > blockCnt=0, waitCnt=108] > at java.lang.Thread.sleep(Native Method) > at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7774) > at > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$WalSegmentSyncer.body(FileWriteAheadLogMana > ger.java:3514) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="wal-file-archiver%null-#46", id=78, state=WAITING, blockCnt=0, > waitCnt=1] > Lock > [object=o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage@79b9388, > ownerName=null, owne > rId=-1] > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at > o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.awaitSegment(SegmentCurrentStateSto > rage.java:73) > at > o.a.i.i.processors.cache.persistence.wal.aware.SegmentCurrentStateStorage.waitNextSegmentForArchivation(Segme > ntCurrentStateStorage.java:90) > at > o.a.i.i.processors.cache.persistence.wal.aware.SegmentAware.waitNextSegmentForArchivation(SegmentAware.java:7 > 9) > at > o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileArchiver.body(FileWriteAheadLogManager. > java:1719) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="rest-#45", id=77, state=TIMED_WAITING, blockCnt=1, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a9e8eed, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="rest-#44", id=76, state=TIMED_WAITING, blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a9e8eed, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="session-timeout-worker-#34", id=51, state=TIMED_WAITING, > blockCnt=0, waitCnt=54] > at java.lang.Thread.sleep(Native Method) > at > o.a.i.i.processors.rest.GridRestProcessor$4.body(GridRestProcessor.java:495) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="exchange-worker-#43", id=74, state=TIMED_WAITING, blockCnt=2, > waitCnt=35] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) > at > o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:219) > at > o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) > at > o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeMan > ager.java:2695) > at > o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeMana > ger.java:2539) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="ignite-update-notifier-timer", id=73, state=TIMED_WAITING, > blockCnt=1, waitCnt=3] > Lock [object=java.util.TaskQueue@d1791fc, ownerName=null, ownerId=-1] > at java.lang.Object.wait(Native Method) > at java.util.TimerThread.mainLoop(Timer.java:552) > at java.util.TimerThread.run(Timer.java:505) > Thread [name="upd-ver-checker", id=72, state=TIMED_WAITING, blockCnt=0, > waitCnt=11] > at java.lang.Thread.sleep(Native Method) > at > o.a.i.i.processors.cluster.GridUpdateNotifier$1.run(GridUpdateNotifier.java:115) > at java.lang.Thread.run(Thread.java:748) > Thread [name="disco-event-worker-#42", id=71, state=WAITING, blockCnt=0, > waitCnt=38] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d62be88, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2856) > at > o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2825) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="tcp-disco-srvr-#3", id=69, state=RUNNABLE, blockCnt=0, > waitCnt=0] > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > at > java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) > - locked java.net.SocksSocketImpl@66be6d7f > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > at java.net.ServerSocket.accept(ServerSocket.java:513) > at > o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5845) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at > o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:5763) > at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > Thread [name="tcp-disco-msg-worker-#2", id=68, state=TIMED_WAITING, > blockCnt=6, waitCnt=5916] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@38097ea3, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) > at > java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) > at > o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7183) > at > o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2700) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at > o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7119) > at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > Thread [name="disco-notifier-worker-#41", id=67, state=WAITING, blockCnt=2, > waitCnt=44] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@920802f, > ownerName=null, ownerId= > -1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at > java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) > at > o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.jav > a:2660) > at > o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java > :2705) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="Thread-9", id=66, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@1849db1a > Thread [name="Thread-8", id=65, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@29a60c27 > Thread [name="Thread-7", id=64, state=RUNNABLE, blockCnt=1, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@79e18e38 > Thread [name="Thread-6", id=63, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@3e2822 > Thread [name="Thread-5", id=62, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@60fa3495 > Thread [name="Thread-4", id=61, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@4602c2a9 > Thread [name="Thread-3", id=60, state=RUNNABLE, blockCnt=1, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@6a62689d > Thread [name="Thread-2", id=59, state=RUNNABLE, blockCnt=1, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at > sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Locked synchronizers: > java.util.concurrent.ThreadPoolExecutor$Worker@4204541c > Thread [name="Thread-1", id=58, state=RUNNABLE, blockCnt=0, waitCnt=0] > at sun.nio.ch.Iocp.getQueuedCompletionStatus(Native Method) > at sun.nio.ch.Iocp.access$300(Iocp.java:46) > at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-data-loader-flusher-#40", id=57, state=WAITING, > blockCnt=0, waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@219ee97d, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2 > 039) > at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) > at > o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-rest-3-#38", id=55, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@345eada4 > - locked java.util.Collections$UnmodifiableSet@475131ea > - locked sun.nio.ch.WindowsSelectorImpl@4a7df0d7 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-rest-2-#37", id=54, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1c7e3de4 > - locked java.util.Collections$UnmodifiableSet@5d8c8042 > - locked sun.nio.ch.WindowsSelectorImpl@131c2478 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-rest-1-#36", id=53, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@36c28584 > - locked java.util.Collections$UnmodifiableSet@5bbe1fa5 > - locked sun.nio.ch.WindowsSelectorImpl@62890a00 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-rest-0-#35", id=52, state=RUNNABLE, > blockCnt=3, waitCnt=1] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@10c4cdcd > - locked java.util.Collections$UnmodifiableSet@328894f4 > - locked sun.nio.ch.WindowsSelectorImpl@1960e637 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="nio-acceptor-tcp-rest-#39", id=56, state=RUNNABLE, blockCnt=0, > waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked sun.nio.ch.Util$3@33132611 > - locked java.util.Collections$UnmodifiableSet@50f2f1b > - locked sun.nio.ch.WindowsSelectorImpl@7356247a > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-client-listener-3-#32", id=49, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6d993579 > - locked java.util.Collections$UnmodifiableSet@4710b575 > - locked sun.nio.ch.WindowsSelectorImpl@3b9b2d7b > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-client-listener-2-#31", id=48, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@79acab86 > - locked java.util.Collections$UnmodifiableSet@5d3a6b37 > - locked sun.nio.ch.WindowsSelectorImpl@1fc3d6f8 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-client-listener-1-#30", id=47, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7dd1c08e > - locked java.util.Collections$UnmodifiableSet@613da429 > - locked sun.nio.ch.WindowsSelectorImpl@6f5018ea > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-client-listener-0-#29", id=46, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6ea577ee > - locked java.util.Collections$UnmodifiableSet@1d830f04 > - locked sun.nio.ch.WindowsSelectorImpl@1113fc71 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="nio-acceptor-client-listener-#33", id=50, state=RUNNABLE, > blockCnt=0, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked sun.nio.ch.Util$3@7e6b31f1 > - locked java.util.Collections$UnmodifiableSet@3bb423d5 > - locked sun.nio.ch.WindowsSelectorImpl@3e4ae088 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="tcp-comm-worker-#1", id=45, state=TIMED_WAITING, blockCnt=0, > waitCnt=1] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30b47344, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4287) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at > o.a.i.spi.communication.tcp.TcpCommunicationSpi$5.body(TcpCommunicationSpi.java:2237) > at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > Thread [name="grid-nio-worker-tcp-comm-3-#27", id=43, state=RUNNABLE, > blockCnt=1, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1777381a > - locked java.util.Collections$UnmodifiableSet@dc70baa > - locked sun.nio.ch.WindowsSelectorImpl@6460f43a > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-comm-2-#26", id=42, state=RUNNABLE, > blockCnt=3, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@775d85a1 > - locked java.util.Collections$UnmodifiableSet@59c4d4b8 > - locked sun.nio.ch.WindowsSelectorImpl@6ceb1e71 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-comm-1-#25", id=41, state=RUNNABLE, > blockCnt=1, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@f49e6f2 > - locked java.util.Collections$UnmodifiableSet@14a20a37 > - locked sun.nio.ch.WindowsSelectorImpl@6bbc7d0f > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-nio-worker-tcp-comm-0-#24", id=40, state=RUNNABLE, > blockCnt=2, waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5f10fe9f > - locked java.util.Collections$UnmodifiableSet@1faeedf9 > - locked sun.nio.ch.WindowsSelectorImpl@1b59af89 > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2151) > at > o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1797) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="nio-acceptor-tcp-comm-#28", id=44, state=RUNNABLE, blockCnt=2, > waitCnt=0] > at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296) > at > sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278) > at > sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159) > at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) > - locked sun.nio.ch.Util$3@1385fde8 > - locked java.util.Collections$UnmodifiableSet@4b390d01 > - locked sun.nio.ch.WindowsSelectorImpl@2390f0ba > at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2929) > at > o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2877) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="grid-timeout-worker-#23", id=39, state=TIMED_WAITING, > blockCnt=3, waitCnt=164] > Lock [object=java.lang.Object@756326a6, ownerName=null, ownerId=-1] > at java.lang.Object.wait(Native Method) > at > o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="ignite-clock", id=38, state=TIMED_WAITING, blockCnt=0, > waitCnt=5617] > at java.lang.Thread.sleep(Native Method) > at o.a.i.i.util.IgniteUtils$22.run(IgniteUtils.java:3379) > at java.lang.Thread.run(Thread.java:748) > Thread [name="pub-#19", id=34, state=TIMED_WAITING, blockCnt=1, waitCnt=2] > Lock > [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33f5d231, > ownerName=null, ownerId > =-1] > at sun.misc.Unsafe.park(Native Method) > at > java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.j > ava:2078) > at > java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) > at > java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Thread [name="jvm-pause-detector-worker", id=31, state=TIMED_WAITING, > blockCnt=0, waitCnt=1360] > at java.lang.Thread.sleep(Native Method) > at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:100) > Thread [name="data-streamer-stripe-7-#16", id=30, state=WAITING, blockCnt=0, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-6-#15", id=29, state=WAITING, blockCnt=1, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-5-#14", id=28, state=WAITING, blockCnt=0, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-4-#13", id=27, state=WAITING, blockCnt=0, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-3-#12", id=26, state=WAITING, blockCnt=3, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-2-#11", id=25, state=WAITING, blockCnt=3, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-1-#10", id=24, state=WAITING, blockCnt=0, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="data-streamer-stripe-0-#9", id=23, state=WAITING, blockCnt=3, > waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-7-#8", id=22, state=WAITING, blockCnt=0, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-6-#7", id=21, state=WAITING, blockCnt=0, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-5-#6", id=20, state=WAITING, blockCnt=0, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-4-#5", id=19, state=WAITING, blockCnt=3, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-3-#4", id=18, state=WAITING, blockCnt=0, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-2-#3", id=17, state=WAITING, blockCnt=3, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-1-#2", id=16, state=WAITING, blockCnt=0, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="sys-stripe-0-#1", id=15, state=WAITING, blockCnt=2, waitCnt=1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) > at > o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:669) > at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:493) > at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) > at java.lang.Thread.run(Thread.java:748) > Thread [name="RMI TCP Accept-0", id=14, state=RUNNABLE, blockCnt=0, waitCnt=0] > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > at > java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) > - locked java.net.SocksSocketImpl@49b5ff5f > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > at java.net.ServerSocket.accept(ServerSocket.java:513) > at > sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) > at java.lang.Thread.run(Thread.java:748) > Thread [name="RMI TCP Accept-49148", id=13, state=RUNNABLE, blockCnt=0, > waitCnt=0] > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > at > java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) > - locked java.net.SocksSocketImpl@4ae7b26d > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > at java.net.ServerSocket.accept(ServerSocket.java:513) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) > at java.lang.Thread.run(Thread.java:748) > Thread [name="RMI TCP Accept-0", id=12, state=RUNNABLE, blockCnt=0, waitCnt=0] > at java.net.DualStackPlainSocketImpl.accept0(Native Method) > at > java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131) > at > java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199) > - locked java.net.SocksSocketImpl@6ed69eab > at java.net.ServerSocket.implAccept(ServerSocket.java:545) > at java.net.ServerSocket.accept(ServerSocket.java:513) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405) > at > sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) > at java.lang.Thread.run(Thread.java:748) > Thread [name="Attach Listener", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0] > Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0] > Thread [name="Finalizer", id=3, state=WAITING, blockCnt=193, waitCnt=5] > Lock [object=java.lang.ref.ReferenceQueue$Lock@2ecc3c1, ownerName=null, > ownerId=-1] > at java.lang.Object.wait(Native Method) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144) > at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165) > at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216) > Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=6, waitCnt=4] > Lock [object=java.lang.ref.Reference$Lock@64c59bc4, ownerName=null, > ownerId=-1] > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:502) > at java.lang.ref.Reference.tryHandlePending(Reference.java:191) > at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) > Thread [name="main", id=1, state=WAITING, blockCnt=5, waitCnt=5] > Lock [object=java.util.concurrent.CountDownLatch$Sync@134629eb, > ownerName=null, ownerId=-1] > at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:8 > 36) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer > .java:997) > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.j > ava:1304) > at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) > at > o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:334) > [18:05:37,502][SEVERE][sys-#62][] JVM will be halted immediately due to the > failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.NullPointerException]] > {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)