[
https://issues.apache.org/jira/browse/IGNITE-29200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29200:
-----------------------------------
Description:
Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
*Root cause*
The test requires two build-idx-runner workers to run concurrently: a
slowed-down forceRebuildIndexes rebuild (10 ms/row over 100k rows) plus a
starting CREATE INDEX whose first processed row releases
slowdownIdxCreateConsumer.startBuildIdxFut.
The build-index thread pool is sized by default as min(4, max(1,
AVAILABLE_PROC_CNT / 4)) (IgniteConfiguration.java:178) and the test never
pinned it. On low-core CPU-limited CI agents this evaluates to 1, so:
1. forceRebuildIndexes starts first and monopolizes the only pool thread for
~1000 s (100k rows × 10 ms).
2. The CREATE INDEX partition worker stays queued in the pool and never starts;
the DDL blocks at GridQueryProcessor.java:2194 (buildIdxFut.get()).
3. The test hangs at ResumeCreateIndexTest.java:252 (startBuildIdxFut.get())
and is killed by the 300 s test timeout. The sleepTime.set(0) speed-up on line
257 is unreachable.
Thread dump at timeout confirms: exactly one build-idx-runner thread in the JVM
executing the rebuild closure, no create-index worker thread.
The sibling testConcurrentBuildNewIndexAndRebuildIndexes0 passes because
create-index starts first and never needs two workers at once — only
...Indexes1's ordering (slow rebuild first) exposes the starvation.
*Fix*
Test-only change (ResumeCreateIndexTest.java): pin the pool the test depends
on: setBuildIndexThreadPoolSize(2) for the node in
testConcurrentBuildNewIndexAndRebuildIndexes1.
Two threads are sufficient — the cache has a single partition
(RendezvousAffinityFunction(false, 1)), i.e. at most two partition workers.
was:
Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
*Root cause*
The test requires two build-idx-runner workers to run concurrently: a
slowed-down forceRebuildIndexes rebuild (10 ms/row over 100k rows) plus a
starting CREATE INDEX whose first processed row releases
slowdownIdxCreateConsumer.startBuildIdxFut.
The build-index thread pool is sized by default as min(4, max(1,
AVAILABLE_PROC_CNT / 4)) (IgniteConfiguration.java:178) and the test never
pinned it. On low-core CPU-limited CI agents this evaluates to 1, so:
1. forceRebuildIndexes starts first and monopolizes the only pool thread for
~1000 s (100k rows × 10 ms).
2. The CREATE INDEX partition worker stays queued in the pool and never starts;
the DDL blocks at GridQueryProcessor.java:2194 (buildIdxFut.get()).
3. The test hangs at ResumeCreateIndexTest.java:252 (startBuildIdxFut.get())
and is killed by the 300 s test timeout. The sleepTime.set(0) speed-up on line
257 is unreachable.
Thread dump at timeout confirms: exactly one build-idx-runner thread in the JVM
executing the rebuild closure, no create-index worker thread.
The sibling testConcurrentBuildNewIndexAndRebuildIndexes0 passes because
create-index starts first and never needs two workers at once — only
...Indexes1's ordering (slow rebuild first) exposes the starvation.
*Fix*
Test-only change (ResumeCreateIndexTest.java):
- Pin the pool the test depends on: setBuildIndexThreadPoolSize(2) for the node
in testConcurrentBuildNewIndexAndRebuildIndexes1. Two threads are sufficient —
the cache has a single partition (RendezvousAffinityFunction(false, 1)), i.e.
at most two partition workers.
> Fix flaky ResumeCreateIndexTest.testConcurrentBuildNewIndexAndRebuildIndexes1
> -----------------------------------------------------------------------------
>
> Key: IGNITE-29200
> URL: https://issues.apache.org/jira/browse/IGNITE-29200
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
> https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
> Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
> out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
> *Root cause*
> The test requires two build-idx-runner workers to run concurrently: a
> slowed-down forceRebuildIndexes rebuild (10 ms/row over 100k rows) plus a
> starting CREATE INDEX whose first processed row releases
> slowdownIdxCreateConsumer.startBuildIdxFut.
> The build-index thread pool is sized by default as min(4, max(1,
> AVAILABLE_PROC_CNT / 4)) (IgniteConfiguration.java:178) and the test never
> pinned it. On low-core CPU-limited CI agents this evaluates to 1, so:
> 1. forceRebuildIndexes starts first and monopolizes the only pool thread for
> ~1000 s (100k rows × 10 ms).
> 2. The CREATE INDEX partition worker stays queued in the pool and never
> starts; the DDL blocks at GridQueryProcessor.java:2194 (buildIdxFut.get()).
> 3. The test hangs at ResumeCreateIndexTest.java:252 (startBuildIdxFut.get())
> and is killed by the 300 s test timeout. The sleepTime.set(0) speed-up on
> line 257 is unreachable.
> Thread dump at timeout confirms: exactly one build-idx-runner thread in the
> JVM executing the rebuild closure, no create-index worker thread.
> The sibling testConcurrentBuildNewIndexAndRebuildIndexes0 passes because
> create-index starts first and never needs two workers at once — only
> ...Indexes1's ordering (slow rebuild first) exposes the starvation.
> *Fix*
> Test-only change (ResumeCreateIndexTest.java): pin the pool the test depends
> on: setBuildIndexThreadPoolSize(2) for the node in
> testConcurrentBuildNewIndexAndRebuildIndexes1.
> Two threads are sufficient — the cache has a single partition
> (RendezvousAffinityFunction(false, 1)), i.e. at most two partition workers.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
