[
https://issues.apache.org/jira/browse/IGNITE-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16495152#comment-16495152
]
Alexey Kuznetsov commented on IGNITE-6538:
------------------------------------------
The exception that led test to hang :
{code:java}
[org.apache.ignite:ignite-core] java.lang.AssertionError:
calculatedOffset=3072, allocated=2048, headerSize=1024
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:346)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:341)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:325)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:167)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:370)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.<init>(MetaStorage.java:472)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:139)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:808)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:901)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[13:46:39]W: [org.apache.ignite:ignite-core] at
java.lang.Thread.run(Thread.java:748)
[13:46:39] : [Step 3/4] [2018-05-22 10:46:39,411][INFO
][exchange-worker-#201982%node0-backUp%][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=6,
minorTopVer=1], resVer=null, err=java.lang.AssertionError:
calculatedOffset=3072, allocated=2048, headerSize=1024]
[13:46:39]W: [org.apache.ignite:ignite-core] [2018-05-22
10:46:39,415][ERROR][exchange-worker-#201982%node0-backUp%][IgniteTestResources]
Critical system error detected. Will be handled accordingly to configured
handler [hnd=class o.a.i.failure.NoOpFailureHandler, failureCtx=FailureContext
[type=SYSTEM_WORKER_TERMINATION, err=java.lang.AssertionError:
calculatedOffset=3072, allocated=2048, headerSize=1024]]
[13:46:39]W: [org.apache.ignite:ignite-core]
java.lang.AssertionError: calculatedOffset=3072, allocated=2048, headerSize=1024
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:346)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:341)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:325)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:779)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:624)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:142)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.freelist.PagesList.init(PagesList.java:167)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.<init>(AbstractFreeList.java:370)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage$FreeListImpl.<init>(MetaStorage.java:472)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.metastorage.MetaStorage.init(MetaStorage.java:139)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:808)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:901)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
[13:46:39]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[13:46:39]W: [org.apache.ignite:ignite-core] at
java.lang.Thread.run(Thread.java:748)
...
Failed to acquire file lock (local nodeId:7ee5a633-993e-4430-8e93-2bfd3484fe9f,
already locked by [effc88b6-b8b5-43a2-8db5-be2e00d6a60a][127.0.0.1,
172.17.0.7][TCP:10802, TCP:45012, TCP:47502]), will try again in 1s:
/data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node2/lock
...
Failed to acquire file lock during 10 sec, (locked by
[effc88b6-b8b5-43a2-8db5-be2e00d6a60a][127.0.0.1, 172.17.0.7][TCP:10802,
TCP:45012, TCP:47502]):
/data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node2/lock
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.tryLock(GridCacheDatabaseSharedManager.java:4700)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lock(GridCacheDatabaseSharedManager.java:922)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:250)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:888)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[13:46:49]W: [org.apache.ignite:ignite-core] at
java.lang.Thread.run(Thread.java:748)
[13:46:49]W: [org.apache.ignite:ignite-core] [2018-05-22
10:46:49,297][ERROR][exchange-worker-#202059%node1-backUp%][GridDhtPartitionsExchangeFuture]
Failed to activate node components
[nodeId=f6f3d53b-7ee3-42c5-add7-9c58066f5f35, client=false,
topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1]]
[13:46:49]W: [org.apache.ignite:ignite-core] class
org.apache.ignite.IgniteCheckedException: Failed to acquire file lock during 10
sec, (locked by [c859c9b3-b57d-44db-bdb8-778302f3d16c][127.0.0.1,
172.17.0.7][TCP:10801, TCP:45011, TCP:47501]):
/data/teamcity/work/c182b70f2dfa6507/work/IgniteChangeGlobalStateTest/db/node1/lock
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$FileLockHolder.tryLock(GridCacheDatabaseSharedManager.java:4700)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.lock(GridCacheDatabaseSharedManager.java:922)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCacheSharedContext.activate(GridCacheSharedContext.java:250)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:888)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
[13:46:49] : [Step 3/4] [2018-05-22 10:46:49,299][INFO
][exchange-worker-#202136%node2-backUp%][GridDhtPartitionsExchangeFuture]
Finished waiting for partition release future [topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=1], waitTime=0ms, futInfo=NA]
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2452)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2332)
[13:46:49]W: [org.apache.ignite:ignite-core] at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[13:46:49]W: [org.apache.ignite:ignite-core] at
java.lang.Thread.run(Thread.java:748)
{code}
> Ignite Activate/Deactivate Cluster suite: After tests validation improvements
> test became flaky on TC
> -----------------------------------------------------------------------------------------------------
>
> Key: IGNITE-6538
> URL: https://issues.apache.org/jira/browse/IGNITE-6538
> Project: Ignite
> Issue Type: Bug
> Components: persistence
> Affects Versions: 2.1
> Reporter: Dmitriy Pavlov
> Assignee: Alexey Kuznetsov
> Priority: Major
> Labels: MakeTeamcityGreenAgain, Muted_test
> Fix For: 2.6
>
>
> https://ci.ignite.apache.org/project.html?projectId=Ignite20Tests&testNameId=-238119157387028099&tab=testDetails&branch_Ignite20Tests=%3Cdefault%3E
> {code}
> [14:36:09]W: [org.apache.ignite:ignite-core] [2018-05-22
> 11:36:09,817][WARN
> ][exchange-worker-#202059%node1-backUp%][GridDhtPartitionsExchangeFuture]
> Unable to await partitions release latch within timeout: ClientLatch
> [coordinator=TcpDiscoveryNode [id=ba7f90d8-3a1f-488a-a8ea-3ae2fdf016fe,
> addrs=[127.0.0.1, 172.17.0.7], sockAddrs=[/127.0.0.1:47503,
> d377975c5791/172.17.0.7:47503], discPort=47503, order=1, intOrder=1,
> lastExchangeTime=1526985998704, loc=false, ver=2.5.0#20180522-sha1:d37470c7,
> isClient=false], ackSent=true, super=CompletableLatch [id=exchange,
> topVer=AffinityTopologyVersion [topVer=6, minorTopVer=1]]]
> {code}
> Fail test:
> {code}
> org.apache.ignite.internal.processors.cache.persistence.standbycluster.IgniteChangeGlobalStateTest#testActivateAfterFailGetLock
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
