[
https://issues.apache.org/jira/browse/IGNITE-29052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29052:
-----------------------------------
Description:
Success Rate: 21.5% (Last 107 Runs) 84 failed, 23 successful
https://ci2.ignite.apache.org/test/7147504629922790672?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Current failure (from CI log _Apache_Ignite_2.x_Tests_Cache_3_43882.log)
The test starts at 15:19:30 and is interrupted after the hardened 600s timeout
(getTestTimeout() = 10 * 60_000 overridden by suite/CI watchdog).
Reported failure:
[ERROR]
org.apache.ignite.internal.processors.cache.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
-- Time elapsed: 601.1 s <<< ERROR!
java.util.concurrent.TimeoutException: Test has been timed out
[test=testRestartsAndCacheCreateDestroy, timeout=600000]
Main thread stack (at dump time, blocked in `.get()`):
o.a.i.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy(IgniteCacheGroupsTest.java:3865)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at jdk.internal.misc.Unsafe.park(Native Method)
Cache destroy thread (`async-runnable-runner-1`) blocked in `destroyCache`:
o.a.i.IgniteCacheGroupsTest$41.run(IgniteCacheGroupsTest.java:3791)
at o.a.i.i.IgniteKernal.destroyCache(IgniteKernal.java:2374)
at o.a.i.i.IgniteKernal.destroyCache0(IgniteKernal.java:2384)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at java.util.concurrent.locks.LockSupport.park(Native Method)
All op threads (`op-thread-3..10`) parked on operation futures, e.g.:
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4120) //
put
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4155) //
putAll
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4145) //
getAll
at
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3906)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at java.util.concurrent.locks.LockSupport.park(Native Method)
*Root cause of the test failure*
IgniteCacheGroupsTest#testRestartsAndCacheCreateDestroy fails not due to test
timeouts or hung transactions, but due to a client-initiated exchange deadlock
(dynamic cache change) racing against a server node restart that rejoins with
the same consistentId/port.
*Mechanism:*
1. The client initiates a dynamic cache change (destroy/create c0) — a
separate distributed exchange (topVer=6.3) started from the client node.
2. At the same time the restart thread stops a server node (node 2,
consistentId=127.0.0.1:47502) and restarts it with the same consistentId/port.
The rejoin spawns its own chain of exchanges (NODE_LEFT, NODE_JOINED, …) which
merge with the client's exchange.
3. The client waits for the coordinator to confirm exchange 6.3 ("waiting
for coordinator response"), but its partition single-messages are never
acknowledged (msgsAckedByRmt=0, recovery descriptor connected=false).
4. Because the client's exchange is never completed, the cluster-wide
merge is blocked; startGrid(2) in the restart thread hangs on await exchange,
and restartFut.get(...) fails with IgniteFutureTimeoutCheckedException at
IgniteCacheGroupsTest.java:3904.
Key detail: the restarted node keeps its identity (consistentId/discPort),
which breaks the client→coordinator recovery session during the rejoin. It's a
race of "client-initiated dynamic-cache-change ⇄ restart of a node with the
same consistentId".
*Proposed fix*
Synchronize client create/destroy and node restarts with a shared ReentrantLock
restartLock in the test:
- restart thread: lock → stopGrid/startGrid → unlock;
- cache thread: lock → destroyCache/createCache → unlock.
A node restart therefore executes sequentially and never overlaps an in-flight
client create/destroy (which is what triggers the deadlock), while ordinary
cache operations continue running concurrently with restarts. The client stays
the initiator of create/destroy, so test coverage is preserved.
A side refinement was made to the group assertion loop to iterate over both
GROUP1/GROUP2 and tolerate a group being absent on a node (all its caches
re-created under another group during the iteration).
was:
Success Rate: 21.5% (Last 107 Runs) 84 failed, 23 successful
https://ci2.ignite.apache.org/test/7147504629922790672?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Current failure (from CI log _Apache_Ignite_2.x_Tests_Cache_3_43882.log)
The test starts at 15:19:30 and is interrupted after the hardened 600s timeout
(getTestTimeout() = 10 * 60_000 overridden by suite/CI watchdog).
Reported failure:
[ERROR]
org.apache.ignite.internal.processors.cache.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
-- Time elapsed: 601.1 s <<< ERROR!
java.util.concurrent.TimeoutException: Test has been timed out
[test=testRestartsAndCacheCreateDestroy, timeout=600000]
Main thread stack (at dump time, blocked in `.get()`):
o.a.i.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy(IgniteCacheGroupsTest.java:3865)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at jdk.internal.misc.Unsafe.park(Native Method)
Cache destroy thread (`async-runnable-runner-1`) blocked in `destroyCache`:
o.a.i.IgniteCacheGroupsTest$41.run(IgniteCacheGroupsTest.java:3791)
at o.a.i.i.IgniteKernal.destroyCache(IgniteKernal.java:2374)
at o.a.i.i.IgniteKernal.destroyCache0(IgniteKernal.java:2384)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at java.util.concurrent.locks.LockSupport.park(Native Method)
All op threads (`op-thread-3..10`) parked on operation futures, e.g.:
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4120) //
put
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4155) //
putAll
o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4145) //
getAll
at
o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3906)
at
o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
at
o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
at java.util.concurrent.locks.LockSupport.park(Native Method)
*Root cause of the test failure*
IgniteCacheGroupsTest#testRestartsAndCacheCreateDestroy fails not due to test
timeouts or hung transactions, but due to a client-initiated exchange deadlock
(dynamic cache change) racing against a server node restart that rejoins with
the same consistentId/port.
*Mechanism:*
1. The client initiates a dynamic cache change (destroy/create c0) — a
separate distributed exchange (topVer=6.3) started from the client node.
2. At the same time the restart thread stops a server node (node 2,
consistentId=127.0.0.1:47502) and restarts it with the same consistentId/port.
The rejoin spawns its own chain of exchanges
(NODE_LEFT, NODE_JOINED, …) which merge with the client's exchange.
3. The client waits for the coordinator to confirm exchange 6.3 ("waiting
for coordinator response"), but its partition single-messages are never
acknowledged (msgsAckedByRmt=0, recovery
descriptor connected=false).
4. Because the client's exchange is never completed, the cluster-wide
merge is blocked; startGrid(2) in the restart thread hangs on await exchange,
and restartFut.get(...) fails with
IgniteFutureTimeoutCheckedException at IgniteCacheGroupsTest.java:3904.
Key detail: the restarted node keeps its identity (consistentId/discPort),
which breaks the client→coordinator recovery session during the rejoin. It's a
race of "client-initiated dynamic-cache-change ⇄ restart of a node with the
same consistentId".
*Proposed fix*
Synchronize client create/destroy and node restarts with a shared ReentrantLock
restartLock in the test:
- restart thread: lock → stopGrid/startGrid → unlock;
- cache thread: lock → destroyCache/createCache → unlock.
A node restart therefore executes sequentially and never overlaps an in-flight
client create/destroy (which is what triggers the deadlock), while ordinary
cache operations continue running concurrently with restarts. The client stays
the initiator of create/destroy, so test coverage is preserved.
A side refinement was made to the group assertion loop to iterate over both
GROUP1/GROUP2 and tolerate a group being absent on a node (all its caches
re-created under another group during the iteration).
> Fix flaky IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
> -----------------------------------------------------------------
>
> Key: IGNITE-29052
> URL: https://issues.apache.org/jira/browse/IGNITE-29052
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Success Rate: 21.5% (Last 107 Runs) 84 failed, 23 successful
> https://ci2.ignite.apache.org/test/7147504629922790672?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
> Current failure (from CI log _Apache_Ignite_2.x_Tests_Cache_3_43882.log)
> The test starts at 15:19:30 and is interrupted after the hardened 600s
> timeout (getTestTimeout() = 10 * 60_000 overridden by suite/CI watchdog).
> Reported failure:
> [ERROR]
> org.apache.ignite.internal.processors.cache.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
> -- Time elapsed: 601.1 s <<< ERROR!
> java.util.concurrent.TimeoutException: Test has been timed out
> [test=testRestartsAndCacheCreateDestroy, timeout=600000]
> Main thread stack (at dump time, blocked in `.get()`):
>
> o.a.i.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy(IgniteCacheGroupsTest.java:3865)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at jdk.internal.misc.Unsafe.park(Native Method)
> Cache destroy thread (`async-runnable-runner-1`) blocked in
> `destroyCache`:
> o.a.i.IgniteCacheGroupsTest$41.run(IgniteCacheGroupsTest.java:3791)
> at o.a.i.i.IgniteKernal.destroyCache(IgniteKernal.java:2374)
> at o.a.i.i.IgniteKernal.destroyCache0(IgniteKernal.java:2384)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at java.util.concurrent.locks.LockSupport.park(Native Method)
> All op threads (`op-thread-3..10`) parked on operation futures, e.g.:
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4120)
> // put
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4155)
> // putAll
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4145)
> // getAll
> at
> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3906)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at java.util.concurrent.locks.LockSupport.park(Native Method)
> *Root cause of the test failure*
> IgniteCacheGroupsTest#testRestartsAndCacheCreateDestroy fails not due to test
> timeouts or hung transactions, but due to a client-initiated exchange
> deadlock (dynamic cache change) racing against a server node restart that
> rejoins with the same consistentId/port.
> *Mechanism:*
> 1. The client initiates a dynamic cache change (destroy/create c0) — a
> separate distributed exchange (topVer=6.3) started from the client node.
> 2. At the same time the restart thread stops a server node (node 2,
> consistentId=127.0.0.1:47502) and restarts it with the same
> consistentId/port. The rejoin spawns its own chain of exchanges (NODE_LEFT,
> NODE_JOINED, …) which merge with the client's exchange.
> 3. The client waits for the coordinator to confirm exchange 6.3
> ("waiting for coordinator response"), but its partition single-messages are
> never acknowledged (msgsAckedByRmt=0, recovery descriptor connected=false).
> 4. Because the client's exchange is never completed, the cluster-wide
> merge is blocked; startGrid(2) in the restart thread hangs on await exchange,
> and restartFut.get(...) fails with IgniteFutureTimeoutCheckedException at
> IgniteCacheGroupsTest.java:3904.
> Key detail: the restarted node keeps its identity (consistentId/discPort),
> which breaks the client→coordinator recovery session during the rejoin. It's
> a race of "client-initiated dynamic-cache-change ⇄ restart of a node with the
> same consistentId".
> *Proposed fix*
> Synchronize client create/destroy and node restarts with a shared
> ReentrantLock restartLock in the test:
> - restart thread: lock → stopGrid/startGrid → unlock;
> - cache thread: lock → destroyCache/createCache → unlock.
> A node restart therefore executes sequentially and never overlaps an
> in-flight client create/destroy (which is what triggers the deadlock), while
> ordinary cache operations continue running concurrently with restarts. The
> client stays the initiator of create/destroy, so test coverage is preserved.
> A side refinement was made to the group assertion loop to iterate over both
> GROUP1/GROUP2 and tolerate a group being absent on a node (all its caches
> re-created under another group during the iteration).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
