joseluisll commented on PR #8577: URL: https://github.com/apache/hbase/pull/8577#issuecomment-6209085615
Some data from a CI failure that hit exactly this, in case it helps the review. It is from #8745 (unrelated, test-only change), `large-wave-1` job: https://github.com/apache/hbase/actions/runs/38062992616/job/116684407597 All 2950 tests passed (33 flakes), but the build failed with `There was a timeout in the fork`. Timeline for the `TestHRegionWithInMemoryFlush` fork: - **~23:12:00** fork starts; the class gets the 13 min `LargeTests` deadline (23:25:00). - **23:25:00** `testParallelIncrementWithMemStoreFlush` (211 s so far) hits the class deadline; the other 30 remaining tests then fail with `TestHRegionWithInMemoryFlush.constructor timed out`. - The executor `shutdownNow()` interrupts the test thread inside `incrementers[i].join()`, so `incrementDone.set(true)` is never reached, and the flusher thread (`Thread-5031`) keeps calling `region.flush(true)` on an empty memstore. - **23:25:01** surefire reruns the failed tests in the same JVM (the rerun gets a fresh 13 min deadline, until 23:38). - **23:27:00** surefire's `forkedProcessTimeoutInSeconds` (900 s) kills the fork, 15 min after it started, while the rerun of `testParallelIncrementWithMemStoreFlush` is still in `join()`. Two things that made it much worse than a plain timeout: 1. **Runaway flusher:** in those ~2.5 minutes the orphaned `Thread-5031` logged **7,975,132** `Flush status journal` lines (one empty flush each) and the test output hit the 1 GiB cap (`Log size exceeded the limit 1073741824, will stop logging`, 23:27:23), competing for CPU and I/O with the rerun. 2. **Unbounded store files:** nothing compacts the standalone region, so every flush adds a file and every increment/append reads all of them. The rerun's flusher (`Thread-6616`) produced **809 store files** in ~2.5 min (405 / 302 / 102 per minute, slowing down as the file count grows). Per-method times in the surefire XML for the in-memory-flush class: `testParallelAppendWithMemStoreFlush` 356 s and `testParallelIncrementWithMemStoreFlush` 257 s, vs. 94 s and 50 s in `TestHRegion`; the class summed to ~912 s against the 780 s budget. So besides splitting the class, it may be worth (a) setting the done flag in a `finally` around the joins so the flusher always stops, and (b) bounding the store file count in the flusher loop (e.g. a `region.compact(true)` every N flushes), otherwise the new classes can still get close to the limit on a slow runner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
