[jira] [Created] (IGNITE-8051) Put operation may hang on LOCAL TRANSACTIONAL cache if it was stopped asynchronously.

[Pavel Pereslegin (JIRA)]

Pavel Pereslegin created IGNITE-8051:
----------------------------------------

             Summary: Put operation may hang on LOCAL TRANSACTIONAL cache if it 
was stopped asynchronously.
                 Key: IGNITE-8051
                 URL: https://issues.apache.org/jira/browse/IGNITE-8051
             Project: Ignite
          Issue Type: Bug
          Components: cache
    Affects Versions: 2.4
            Reporter: Pavel Pereslegin


Put operation may hang if local cache was destroyed asynchronously. 
It seems that this happens because 
IgniteTxImplicitSingleStateImpl#topologyReadLock does not check cache status 
for local cache. 

Simple reproducer: 
{code:java}
public class DestroyLocalCacheTest extends GridCommonAbstractTest {
    /** */
    public void testDestroyAsync() throws Exception {
        try (Ignite node = startGrid()) {
            IgniteCache<Integer, Boolean> locCache = node.createCache(
                new CacheConfiguration<Integer, 
Boolean>(DEFAULT_CACHE_NAME).setCacheMode(CacheMode.LOCAL)
                    .setAtomicityMode(CacheAtomicityMode.TRANSACTIONAL));

            AtomicInteger cntr = new AtomicInteger();

            GridTestUtils.runMultiThreadedAsync(() -> {
                try {
                    int key;

                    while ((key = cntr.getAndIncrement()) < 10_000) {

                        if (key == 1000)
                            locCache.destroy();

                        locCache.put(key, true);
                    }
                }
                catch (Exception ignore) {
                    // No-op
                }

                return null;
            }, 5, "put-thread").get();
        }
    }
}
{code}

Log output:
{noformat}
[2018-03-26 19:10:11,350][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridCacheProcessor] Started 
cache [name=default, id=1544803905, memoryPolicyName=default, mode=LOCAL, 
atomicity=TRANSACTIONAL, backups=0]
[2018-03-26 19:10:11,352][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 Finished waiting for partition release future [topVer=AffinityTopologyVersion 
[topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA]
[2018-03-26 19:10:11,353][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]]
[2018-03-26 19:10:11,354][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], 
err=null]
[2018-03-26 19:10:11,354][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][time] Finished exchange 
init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true]
[2018-03-26 19:10:11,355][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridCachePartitionExchangeManager]
 Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion 
[topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, 
node=fe867235-2fff-465a-b767-206abd291058]
[2018-03-26 19:10:11,590][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][time] Started exchange init 
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], crd=true, 
evt=DISCOVERY_CUSTOM_EVT, evtNode=fe867235-2fff-465a-b767-206abd291058, 
customEvt=DynamicCacheChangeBatch 
[id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, 
reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, 
nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, 
destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions 
[startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, 
destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], 
allowMerge=false]
[2018-03-26 19:10:11,590][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 Finished waiting for partition release future [topVer=AffinityTopologyVersion 
[topVer=1, minorTopVer=2], waitTime=0ms, futInfo=NA]
[2018-03-26 19:10:11,591][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=2], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=2]]
[2018-03-26 19:10:11,592][INFO 
][exchange-worker-#38%local.DestroyLocalCacheTest%][GridDhtPartitionsExchangeFuture]
 Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=2], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], 
err=null]
[2018-03-26 19:15:09,191][ERROR][main][root] Test has been timed out and will 
be interrupted (threads dump will be taken before interruption) 
[test=testDestroyAsync, timeout=300000]
[2018-03-26 19:15:09,191][WARN ][main][diagnostic] Dumping debug info for node 
[id=fe867235-2fff-465a-b767-206abd291058, name=local.DestroyLocalCacheTest, 
order=1, topVer=1, client=false]
[2018-03-26 19:15:09,191][WARN ][main][diagnostic] Ready affinity version: 
AffinityTopologyVersion [topVer=1, minorTopVer=1]
[2018-03-26 19:15:09,195][WARN ][main][diagnostic] Last exchange future: 
GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent 
[customMsg=DynamicCacheChangeBatch 
[id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, 
reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, 
nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, 
destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions 
[startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, 
destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], 
affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], 
super=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], topVer=1, nodeId8=fe867235, msg=null, 
type=DISCOVERY_CUSTOM_EVT, tstamp=1522080611585]], crd=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], exchId=GridDhtPartitionExchangeId 
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], 
discoEvt=DiscoveryCustomEvent [customMsg=DynamicCacheChangeBatch 
[id=67313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, 
reqs=[DynamicCacheChangeRequest [cacheName=default, hasCfg=false, 
nodeId=fe867235-2fff-465a-b767-206abd291058, clientStartOnly=false, stop=true, 
destroy=false, disabledAfterStartfalse]], exchangeActions=ExchangeActions 
[startCaches=null, stopCaches=[default], startGrps=[], stopGrps=[default, 
destroy=true], resetParts=null, stateChangeRequest=null], startCaches=false], 
affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], 
super=DiscoveryEvent [evtNode=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], topVer=1, nodeId8=fe867235, msg=null, 
type=DISCOVERY_CUSTOM_EVT, tstamp=1522080611585]], nodeId=fe867235, 
evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter 
[ignoreInterrupts=false, state=INIT, res=null, hash=13648335], init=false, 
lastVer=GridCacheVersion [topVer=133560612, order=1522080613658, nodeOrder=1], 
partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion 
[topVer=1, minorTopVer=2], futures=[ExplicitLockReleaseFuture 
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]], 
TxReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], 
futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion 
[topVer=1, minorTopVer=2], futures=[]], DataStreamerReleaseFuture 
[topVer=AffinityTopologyVersion [topVer=1, minorTopVer=2], futures=[]]]], 
exchActions=ExchangeActions [startCaches=null, stopCaches=[default], 
startGrps=[], stopGrps=[default, destroy=true], resetParts=null, 
stateChangeRequest=null], affChangeMsg=null, initTs=1522080611585, 
centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null, 
done=true, state=DONE, evtLatch=0, remaining=[], super=GridFutureAdapter 
[ignoreInterrupts=false, state=INIT, res=null, hash=747464370]]
[2018-03-26 19:15:09,196][WARN ][main][GridCachePartitionExchangeManager] First 
10 pending exchange futures [total=0]
[2018-03-26 19:15:09,197][WARN ][main][diagnostic] Last 10 exchange futures 
(total: 3):
[2018-03-26 19:15:09,197][WARN ][main][diagnostic] >>> 
GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=2], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], done=false]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] >>> 
GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], done=true]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] >>> 
GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, 
minorTopVer=0], evt=NODE_JOINED, evtNode=TcpDiscoveryNode 
[id=fe867235-2fff-465a-b767-206abd291058, addrs=[127.0.0.1], 
sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, 
lastExchangeTime=1522080609915, loc=true, ver=2.5.0#19700101-sha1:00000000, 
isClient=false], done=true]
[2018-03-26 19:15:09,198][WARN ][main][diagnostic] Pending transactions:
[2018-03-26 19:15:09,200][WARN ][main][diagnostic] >>> [txVer=null, 
tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], 
nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, 
hasRemoteLocks=false, trackTimeout=false, thread=put-thread-1, 
mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter 
[nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, 
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, 
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, 
recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion 
[topVer=133560612, order=1522080613657, nodeOrder=1], writeVer=null, 
implicit=true, loc=true, threadId=62, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613657, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297604ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,200][WARN ][main][diagnostic] >>> [txVer=null, 
tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], 
nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, 
hasRemoteLocks=false, trackTimeout=false, thread=put-thread-2, 
mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter 
[nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, 
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, 
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, 
recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion 
[topVer=133560612, order=1522080613653, nodeOrder=1], writeVer=null, 
implicit=true, loc=true, threadId=63, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613653, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] >>> [txVer=null, 
tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], 
nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, 
hasRemoteLocks=false, trackTimeout=false, thread=put-thread-5, 
mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter 
[nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, 
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, 
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, 
recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion 
[topVer=133560612, order=1522080613652, nodeOrder=1], writeVer=null, 
implicit=true, loc=true, threadId=66, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613652, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] >>> [txVer=null, 
tx=GridNearTxLocal [mappings=IgniteTxMappingsSingleImpl [mapping=null], 
nearLocallyMapped=false, colocatedLocallyMapped=false, needCheckBackup=null, 
hasRemoteLocks=false, trackTimeout=false, thread=put-thread-4, 
mappings=IgniteTxMappingsSingleImpl [mapping=null], super=GridDhtTxLocalAdapter 
[nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false, 
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false, 
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true, 
recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion 
[topVer=133560612, order=1522080613645, nodeOrder=1], writeVer=null, 
implicit=true, loc=true, threadId=65, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613645, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], size=1]]]]
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] Pending explicit locks:
[2018-03-26 19:15:09,201][WARN ][main][diagnostic] Pending cache futures:
[2018-03-26 19:15:09,203][WARN ][main][diagnostic] >>> GridNearTxFinishFuture 
[futId=8c313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal 
[mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, 
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, 
trackTimeout=false, thread=put-thread-5, mappings=IgniteTxMappingsSingleImpl 
[mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, 
nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter 
[completedBase=null, sndTransformedVals=false, depEnabled=false, 
txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], 
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, 
order=1522080613652, nodeOrder=1], writeVer=null, implicit=true, loc=true, 
threadId=66, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613652, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], 
size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], 
trackable=true, finishOnePhaseCalled=false, innerFuts=[], 
super=GridCompoundIdentityFuture [super=GridCompoundFuture 
[rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, 
cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,203][WARN ][main][diagnostic] >>> GridNearTxFinishFuture 
[futId=fc313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal 
[mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, 
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, 
trackTimeout=false, thread=put-thread-1, mappings=IgniteTxMappingsSingleImpl 
[mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, 
nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter 
[completedBase=null, sndTransformedVals=false, depEnabled=false, 
txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], 
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, 
order=1522080613657, nodeOrder=1], writeVer=null, implicit=true, loc=true, 
threadId=62, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613657, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], 
size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], 
trackable=true, finishOnePhaseCalled=false, innerFuts=[], 
super=GridCompoundIdentityFuture [super=GridCompoundFuture 
[rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, 
cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] >>> GridNearTxFinishFuture 
[futId=cb313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal 
[mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, 
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, 
trackTimeout=false, thread=put-thread-4, mappings=IgniteTxMappingsSingleImpl 
[mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, 
nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter 
[completedBase=null, sndTransformedVals=false, depEnabled=false, 
txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], 
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, 
order=1522080613645, nodeOrder=1], writeVer=null, implicit=true, loc=true, 
threadId=65, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613645, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], 
size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], 
trackable=true, finishOnePhaseCalled=false, innerFuts=[], 
super=GridCompoundIdentityFuture [super=GridCompoundFuture 
[rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, 
cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] >>> GridNearTxFinishFuture 
[futId=cc313136261-d40acb94-363f-4c5e-b62b-1a95fbe73164, tx=GridNearTxLocal 
[mappings=IgniteTxMappingsSingleImpl [mapping=null], nearLocallyMapped=false, 
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false, 
trackTimeout=false, thread=put-thread-2, mappings=IgniteTxMappingsSingleImpl 
[mapping=null], super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, 
nearNodes=[], dhtNodes=[], explicitLock=false, super=IgniteTxLocalAdapter 
[completedBase=null, sndTransformedVals=false, depEnabled=false, 
txState=IgniteTxImplicitSingleStateImpl [init=true, recovery=false], 
super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=133560612, 
order=1522080613653, nodeOrder=1], writeVer=null, implicit=true, loc=true, 
threadId=63, startTime=1522080611585, 
nodeId=fe867235-2fff-465a-b767-206abd291058, startVer=GridCacheVersion 
[topVer=133560612, order=1522080613653, nodeOrder=1], endVer=null, 
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0, 
sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=NONE, 
invalidParts=null, state=ACTIVE, timedOut=false, topVer=AffinityTopologyVersion 
[topVer=-1, minorTopVer=0], duration=297614ms, onePhaseCommit=false], 
size=1]]], commit=true, mappings=IgniteTxMappingsSingleImpl [mapping=null], 
trackable=true, finishOnePhaseCalled=false, innerFuts=[], 
super=GridCompoundIdentityFuture [super=GridCompoundFuture 
[rdc=AlwaysTrueReducer [], initFlag=0, lsnrCalls=0, done=false, 
cancelled=false, err=null, futs=[]]]]
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] Pending atomic cache futures:
[2018-03-26 19:15:09,204][WARN ][main][diagnostic] Pending data streamer 
futures:
[2018-03-26 19:15:09,205][WARN ][main][diagnostic] Pending transaction deadlock 
detection futures:
[19:15:09] (wrn) No deadlocked threads detected.

[19:15:09] (wrn) Thread dump at 2018/03/26 19:15:09 MSK[2018-03-26 
19:15:09,226][WARN ][main][root] No deadlocked threads detected.
Thread [name="sys-#69%local.DestroyLocalCacheTest%", id=95, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#68%local.DestroyLocalCacheTest%", id=94, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#67%local.DestroyLocalCacheTest%", id=93, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#66%local.DestroyLocalCacheTest%", id=92, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#65%local.DestroyLocalCacheTest%", id=91, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#64%local.DestroyLocalCacheTest%", id=90, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="put-thread-5", id=66, state=WAITING, blockCnt=138, waitCnt=61]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-4", id=65, state=WAITING, blockCnt=109, waitCnt=43]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-3", id=64, state=WAITING, blockCnt=93, waitCnt=56]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.destroy(GatewayProtectedCacheProxy.java:1503)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:30)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-2", id=63, state=WAITING, blockCnt=132, waitCnt=57]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-1", id=62, state=WAITING, blockCnt=135, waitCnt=76]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="async-callable-runner-1", id=61, state=WAITING, blockCnt=0, 
waitCnt=1]
    Lock [object=o.a.i.testframework.GridTestSafeThreadFactory$1@1cd072a9, 
ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at 
o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:871)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:811)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:809)
        at o.a.i.testframework.GridTestUtils$7.run(GridTestUtils.java:979)
        at o.a.i.testframework.GridTestUtils$9.call(GridTestUtils.java:1275)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="ttl-cleanup-worker-#40%local.DestroyLocalCacheTest%", id=60, 
state=TIMED_WAITING, blockCnt=0, waitCnt=596]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7598)
        at 
o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="srvc-deploy-#39%local.DestroyLocalCacheTest%", id=58, 
state=WAITING, blockCnt=0, waitCnt=3]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c75222b,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#38%local.DestroyLocalCacheTest%", id=57, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1489]
    Lock 
[object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@48140564, 
ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at 
o.a.i.i.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:201)
        at 
o.a.i.i.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:305)
        at 
o.a.i.i.processors.cache.GridCacheProcessor.stopGateway(GridCacheProcessor.java:2060)
        at 
o.a.i.i.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2229)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1612)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:2610)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:2408)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1135)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:704)
        at 
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2344)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4c203ea1
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@27f674d
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@1d251891
Thread [name="disco-event-worker-#37%local.DestroyLocalCacheTest%", id=56, 
state=WAITING, blockCnt=0, waitCnt=299]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ceff1,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2598)
        at 
o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2580)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-ip-finder-cleaner-#6%local.DestroyLocalCacheTest%", 
id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
        at java.lang.Thread.sleep(Native Method)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1883)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-multicast-addr-sender-#5%local.DestroyLocalCacheTest%", 
id=54, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at 
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked java.net.DatagramPacket@49e4cb85
        - locked java.net.MulticastSocket@2133c8f8
        at 
o.a.i.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder$AddressSender.body(TcpDiscoveryMulticastIpFinder.java:882)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-srvr-#4%local.DestroyLocalCacheTest%", id=53, 
state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5590)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#3%local.DestroyLocalCacheTest%", id=52, 
state=TIMED_WAITING, blockCnt=4, waitCnt=29402]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43a25848,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at 
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6702)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2602)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-data-loader-flusher-#36%local.DestroyLocalCacheTest%", 
id=51, state=WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ac3fd8b,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at 
o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-3-#34%local.DestroyLocalCacheTest%", 
id=49, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5594a1b5
        - locked java.util.Collections$UnmodifiableSet@6a5fc7f7
        - locked sun.nio.ch.EPollSelectorImpl@3b6eb2ec
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-2-#33%local.DestroyLocalCacheTest%", 
id=48, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1e643faf
        - locked java.util.Collections$UnmodifiableSet@6e8dacdf
        - locked sun.nio.ch.EPollSelectorImpl@7a79be86
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-1-#32%local.DestroyLocalCacheTest%", 
id=47, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@34ce8af7
        - locked java.util.Collections$UnmodifiableSet@b684286
        - locked sun.nio.ch.EPollSelectorImpl@880ec60
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-0-#31%local.DestroyLocalCacheTest%", 
id=46, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3f3afe78
        - locked java.util.Collections$UnmodifiableSet@7f63425a
        - locked sun.nio.ch.EPollSelectorImpl@36d64342
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#30%local.DestroyLocalCacheTest%", id=45, 
state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@39ba5a14
        - locked java.util.Collections$UnmodifiableSet@511baa65
        - locked sun.nio.ch.EPollSelectorImpl@340f438e
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-sharedfs-timeout-worker-#2%local.DestroyLocalCacheTest%", 
id=44, state=TIMED_WAITING, blockCnt=0, waitCnt=60]
    Lock [object=java.lang.Object@30c7da1e, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at 
o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-comm-worker-#1%local.DestroyLocalCacheTest%", id=43, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b464ce8,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4093)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-3-#29%local.DestroyLocalCacheTest%", 
id=42, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@57829d67
        - locked java.util.Collections$UnmodifiableSet@19dfb72a
        - locked sun.nio.ch.EPollSelectorImpl@17c68925
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#28%local.DestroyLocalCacheTest%", 
id=41, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7e0ea639
        - locked java.util.Collections$UnmodifiableSet@3d24753a
        - locked sun.nio.ch.EPollSelectorImpl@59a6e353
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#27%local.DestroyLocalCacheTest%", 
id=40, state=RUNNABLE, blockCnt=2, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7a0ac6e3
        - locked java.util.Collections$UnmodifiableSet@71be98f5
        - locked sun.nio.ch.EPollSelectorImpl@6fadae5d
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#26%local.DestroyLocalCacheTest%", 
id=39, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17f6480
        - locked java.util.Collections$UnmodifiableSet@2d6e8792
        - locked sun.nio.ch.EPollSelectorImpl@2812cbfa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#25%local.DestroyLocalCacheTest%", id=38, 
state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@2acf57e3
        - locked java.util.Collections$UnmodifiableSet@506e6d5e
        - locked sun.nio.ch.EPollSelectorImpl@96532d6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#24%local.DestroyLocalCacheTest%", id=37, 
state=TIMED_WAITING, blockCnt=1, waitCnt=956]
    Lock [object=java.lang.Object@3796751b, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at 
o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#17%local.DestroyLocalCacheTest%", id=30, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#16%local.DestroyLocalCacheTest%", id=29, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#15%local.DestroyLocalCacheTest%", id=28, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#14%local.DestroyLocalCacheTest%", id=27, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#13%local.DestroyLocalCacheTest%", id=26, 
state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#12%local.DestroyLocalCacheTest%", id=25, 
state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#11%local.DestroyLocalCacheTest%", id=24, 
state=WAITING, blockCnt=2, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#10%local.DestroyLocalCacheTest%", id=23, 
state=WAITING, blockCnt=1, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#9%local.DestroyLocalCacheTest%", id=22, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#8%local.DestroyLocalCacheTest%", id=21, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#7%local.DestroyLocalCacheTest%", id=20, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#6%local.DestroyLocalCacheTest%", id=19, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#5%local.DestroyLocalCacheTest%", id=18, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#4%local.DestroyLocalCacheTest%", id=17, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#3%local.DestroyLocalCacheTest%", id=16, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#2%local.DestroyLocalCacheTest%", id=15, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="test-runner-#1%local.DestroyLocalCacheTest%", id=14, 
state=WAITING, blockCnt=4, waitCnt=5]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.testDestroyAsync(DestroyLocalCacheTest.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at junit.framework.TestCase.runTest(TestCase.java:176)
        at 
o.a.i.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
        at 
o.a.i.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
        at 
o.a.i.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=13, state=TIMED_WAITING, 
blockCnt=0, waitCnt=5984]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

Thread [name="ignite-clock-for-tests", id=11, state=TIMED_WAITING, blockCnt=0, 
waitCnt=29548]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.GridTestClockTimer.run(GridTestClockTimer.java:41)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Monitor Ctrl-Break", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at 
com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name="Finalizer", id=3, state=WAITING, blockCnt=3, waitCnt=4]
    Lock [object=java.lang.ref.ReferenceQueue$Lock@4411d970, ownerName=null, 
ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=3, waitCnt=3]
    Lock [object=java.lang.ref.Reference$Lock@6442b0a6, ownerName=null, 
ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1324)
        at 
o.a.i.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1942)
        at junit.framework.TestCase.runBare(TestCase.java:141)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:129)
        at junit.framework.TestSuite.runTest(TestSuite.java:255)
        at junit.framework.TestSuite.run(TestSuite.java:250)
        at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
        at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
        at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
        at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)



[2018-03-26 19:15:09,244][WARN ][main][root] Thread dump at 2018/03/26 19:15:09 
MSK
Thread [name="sys-#69%local.DestroyLocalCacheTest%", id=95, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#68%local.DestroyLocalCacheTest%", id=94, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#67%local.DestroyLocalCacheTest%", id=93, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#66%local.DestroyLocalCacheTest%", id=92, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#65%local.DestroyLocalCacheTest%", id=91, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-#64%local.DestroyLocalCacheTest%", id=90, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e580929,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="put-thread-5", id=66, state=WAITING, blockCnt=138, waitCnt=61]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-4", id=65, state=WAITING, blockCnt=109, waitCnt=43]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-3", id=64, state=WAITING, blockCnt=93, waitCnt=56]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at o.a.i.i.util.future.IgniteFutureImpl.get(IgniteFutureImpl.java:134)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.destroy(GatewayProtectedCacheProxy.java:1503)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:30)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-2", id=63, state=WAITING, blockCnt=132, waitCnt=57]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="put-thread-1", id=62, state=WAITING, blockCnt=135, waitCnt=76]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2390)
        at 
o.a.i.i.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4088)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2388)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2369)
        at 
o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2346)
        at 
o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1084)
        at 
o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:886)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.lambda$testDestroyAsync$0(DestroyLocalCacheTest.java:32)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest$$Lambda$7/759487471.call(Unknown
 Source)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="async-callable-runner-1", id=61, state=WAITING, blockCnt=0, 
waitCnt=1]
    Lock [object=o.a.i.testframework.GridTestSafeThreadFactory$1@1cd072a9, 
ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Thread.join(Thread.java:1252)
        at java.lang.Thread.join(Thread.java:1326)
        at 
o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:871)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:811)
        at o.a.i.testframework.GridTestUtils$4.call(GridTestUtils.java:809)
        at o.a.i.testframework.GridTestUtils$7.run(GridTestUtils.java:979)
        at o.a.i.testframework.GridTestUtils$9.call(GridTestUtils.java:1275)
        at o.a.i.testframework.GridTestThread.run(GridTestThread.java:86)

Thread [name="ttl-cleanup-worker-#40%local.DestroyLocalCacheTest%", id=60, 
state=TIMED_WAITING, blockCnt=0, waitCnt=596]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7598)
        at 
o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:137)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="srvc-deploy-#39%local.DestroyLocalCacheTest%", id=58, 
state=WAITING, blockCnt=0, waitCnt=3]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c75222b,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="exchange-worker-#38%local.DestroyLocalCacheTest%", id=57, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1489]
    Lock 
[object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@48140564, 
ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:934)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1247)
        at 
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115)
        at 
o.a.i.i.util.StripedCompositeReadWriteLock$WriteLock.tryLock(StripedCompositeReadWriteLock.java:201)
        at 
o.a.i.i.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:305)
        at 
o.a.i.i.processors.cache.GridCacheProcessor.stopGateway(GridCacheProcessor.java:2060)
        at 
o.a.i.i.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2229)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:1612)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:2610)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:2408)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1135)
        at 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:704)
        at 
o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2344)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

    Locked synchronizers:
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4c203ea1
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@27f674d
        java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@1d251891
Thread [name="disco-event-worker-#37%local.DestroyLocalCacheTest%", id=56, 
state=WAITING, blockCnt=0, waitCnt=299]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ceff1,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at 
o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2598)
        at 
o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2580)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="tcp-disco-ip-finder-cleaner-#6%local.DestroyLocalCacheTest%", 
id=55, state=TIMED_WAITING, blockCnt=0, waitCnt=9]
        at java.lang.Thread.sleep(Native Method)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1883)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-multicast-addr-sender-#5%local.DestroyLocalCacheTest%", 
id=54, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainDatagramSocketImpl.receive0(Native Method)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at 
java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:143)
        - locked java.net.PlainDatagramSocketImpl@7c30a502
        at java.net.DatagramSocket.receive(DatagramSocket.java:812)
        - locked java.net.DatagramPacket@49e4cb85
        - locked java.net.MulticastSocket@2133c8f8
        at 
o.a.i.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder$AddressSender.body(TcpDiscoveryMulticastIpFinder.java:882)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-srvr-#4%local.DestroyLocalCacheTest%", id=53, 
state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at 
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
        at java.net.ServerSocket.implAccept(ServerSocket.java:545)
        at java.net.ServerSocket.accept(ServerSocket.java:513)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5590)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-disco-msg-worker-#3%local.DestroyLocalCacheTest%", id=52, 
state=TIMED_WAITING, blockCnt=4, waitCnt=29404]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@43a25848,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
        at 
java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerAdapter.body(ServerImpl.java:6702)
        at 
o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2602)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-data-loader-flusher-#36%local.DestroyLocalCacheTest%", 
id=51, state=WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ac3fd8b,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:211)
        at 
o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:108)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-3-#34%local.DestroyLocalCacheTest%", 
id=49, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5594a1b5
        - locked java.util.Collections$UnmodifiableSet@6a5fc7f7
        - locked sun.nio.ch.EPollSelectorImpl@3b6eb2ec
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-2-#33%local.DestroyLocalCacheTest%", 
id=48, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1e643faf
        - locked java.util.Collections$UnmodifiableSet@6e8dacdf
        - locked sun.nio.ch.EPollSelectorImpl@7a79be86
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-1-#32%local.DestroyLocalCacheTest%", 
id=47, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@34ce8af7
        - locked java.util.Collections$UnmodifiableSet@b684286
        - locked sun.nio.ch.EPollSelectorImpl@880ec60
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread 
[name="grid-nio-worker-client-listener-0-#31%local.DestroyLocalCacheTest%", 
id=46, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3f3afe78
        - locked java.util.Collections$UnmodifiableSet@7f63425a
        - locked sun.nio.ch.EPollSelectorImpl@36d64342
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#30%local.DestroyLocalCacheTest%", id=45, 
state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@39ba5a14
        - locked java.util.Collections$UnmodifiableSet@511baa65
        - locked sun.nio.ch.EPollSelectorImpl@340f438e
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-sharedfs-timeout-worker-#2%local.DestroyLocalCacheTest%", 
id=44, state=TIMED_WAITING, blockCnt=0, waitCnt=60]
    Lock [object=java.lang.Object@30c7da1e, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at 
o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="tcp-comm-worker-#1%local.DestroyLocalCacheTest%", id=43, 
state=TIMED_WAITING, blockCnt=0, waitCnt=1]
    Lock 
[object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b464ce8,
 ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at 
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at 
o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4093)
        at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)

Thread [name="grid-nio-worker-tcp-comm-3-#29%local.DestroyLocalCacheTest%", 
id=42, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@57829d67
        - locked java.util.Collections$UnmodifiableSet@19dfb72a
        - locked sun.nio.ch.EPollSelectorImpl@17c68925
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-2-#28%local.DestroyLocalCacheTest%", 
id=41, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7e0ea639
        - locked java.util.Collections$UnmodifiableSet@3d24753a
        - locked sun.nio.ch.EPollSelectorImpl@59a6e353
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-1-#27%local.DestroyLocalCacheTest%", 
id=40, state=RUNNABLE, blockCnt=2, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7a0ac6e3
        - locked java.util.Collections$UnmodifiableSet@71be98f5
        - locked sun.nio.ch.EPollSelectorImpl@6fadae5d
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-nio-worker-tcp-comm-0-#26%local.DestroyLocalCacheTest%", 
id=39, state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked o.a.i.i.util.nio.SelectedSelectionKeySet@17f6480
        - locked java.util.Collections$UnmodifiableSet@2d6e8792
        - locked sun.nio.ch.EPollSelectorImpl@2812cbfa
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2075)
        at 
o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1749)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="nio-acceptor-#25%local.DestroyLocalCacheTest%", id=38, 
state=RUNNABLE, blockCnt=1, waitCnt=0]
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked sun.nio.ch.Util$3@2acf57e3
        - locked java.util.Collections$UnmodifiableSet@506e6d5e
        - locked sun.nio.ch.EPollSelectorImpl@96532d6
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2813)
        at 
o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2785)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="grid-timeout-worker-#24%local.DestroyLocalCacheTest%", id=37, 
state=TIMED_WAITING, blockCnt=1, waitCnt=956]
    Lock [object=java.lang.Object@3796751b, ownerName=null, ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at 
o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:195)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:110)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-7-#17%local.DestroyLocalCacheTest%", id=30, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-6-#16%local.DestroyLocalCacheTest%", id=29, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-5-#15%local.DestroyLocalCacheTest%", id=28, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-4-#14%local.DestroyLocalCacheTest%", id=27, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-3-#13%local.DestroyLocalCacheTest%", id=26, 
state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-2-#12%local.DestroyLocalCacheTest%", id=25, 
state=WAITING, blockCnt=3, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-1-#11%local.DestroyLocalCacheTest%", id=24, 
state=WAITING, blockCnt=2, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="data-streamer-stripe-0-#10%local.DestroyLocalCacheTest%", id=23, 
state=WAITING, blockCnt=1, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-7-#9%local.DestroyLocalCacheTest%", id=22, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-6-#8%local.DestroyLocalCacheTest%", id=21, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-5-#7%local.DestroyLocalCacheTest%", id=20, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-4-#6%local.DestroyLocalCacheTest%", id=19, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-3-#5%local.DestroyLocalCacheTest%", id=18, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-2-#4%local.DestroyLocalCacheTest%", id=17, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-1-#3%local.DestroyLocalCacheTest%", id=16, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="sys-stripe-0-#2%local.DestroyLocalCacheTest%", id=15, 
state=WAITING, blockCnt=0, waitCnt=1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:651)
        at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="test-runner-#1%local.DestroyLocalCacheTest%", id=14, 
state=WAITING, blockCnt=4, waitCnt=5]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
        at 
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
        at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
        at 
o.a.i.i.processors.cache.local.DestroyLocalCacheTest.testDestroyAsync(DestroyLocalCacheTest.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at junit.framework.TestCase.runTest(TestCase.java:176)
        at 
o.a.i.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
        at 
o.a.i.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
        at 
o.a.i.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="jvm-pause-detector-worker", id=13, state=TIMED_WAITING, 
blockCnt=0, waitCnt=5984]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:90)

Thread [name="ignite-clock-for-tests", id=11, state=TIMED_WAITING, blockCnt=0, 
waitCnt=29550]
        at java.lang.Thread.sleep(Native Method)
        at o.a.i.i.util.GridTestClockTimer.run(GridTestClockTimer.java:41)
        at java.lang.Thread.run(Thread.java:748)

Thread [name="Monitor Ctrl-Break", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0]
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked java.io.InputStreamReader@67b64c45
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at 
com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64)

Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0]

Thread [name="Finalizer", id=3, state=WAITING, blockCnt=3, waitCnt=4]
    Lock [object=java.lang.ref.ReferenceQueue$Lock@4411d970, ownerName=null, 
ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=3, waitCnt=3]
    Lock [object=java.lang.ref.Reference$Lock@6442b0a6, ownerName=null, 
ownerId=-1]
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

Thread [name="main", id=1, state=RUNNABLE, blockCnt=0, waitCnt=1]
        at sun.management.ThreadImpl.dumpThreads0(Native Method)
        at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
        at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1324)
        at 
o.a.i.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:1944)
        at junit.framework.TestCase.runBare(TestCase.java:141)
        at junit.framework.TestResult$1.protect(TestResult.java:122)
        at junit.framework.TestResult.runProtected(TestResult.java:142)
        at junit.framework.TestResult.run(TestResult.java:125)
        at junit.framework.TestCase.run(TestCase.java:129)
        at junit.framework.TestSuite.runTest(TestSuite.java:255)
        at junit.framework.TestSuite.run(TestSuite.java:250)
        at 
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:160)
        at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
        at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
        at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
        at 
com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)