Sigma-Ma commented on PR #8577: URL: https://github.com/apache/hbase/pull/8577#issuecomment-6209815746
> Some data from a CI failure that hit exactly this, in case it helps the review. It is from #8745 (unrelated, test-only change), `large-wave-1` job: https://github.com/apache/hbase/actions/runs/38062992616/job/116684407597 > > All 2950 tests passed (33 flakes), but the build failed with `There was a timeout in the fork`. Timeline for the `TestHRegionWithInMemoryFlush` fork: > > * **~23:12:00** fork starts; the class gets the 13 min `LargeTests` deadline (23:25:00). > * **23:25:00** `testParallelIncrementWithMemStoreFlush` (211 s so far) hits the class deadline; the other 30 remaining tests then fail with `TestHRegionWithInMemoryFlush.constructor timed out`. > * The executor `shutdownNow()` interrupts the test thread inside `incrementers[i].join()`, so `incrementDone.set(true)` is never reached, and the flusher thread (`Thread-5031`) keeps calling `region.flush(true)` on an empty memstore. > * **23:25:01** surefire reruns the failed tests in the same JVM (the rerun gets a fresh 13 min deadline, until 23:38). > * **23:27:00** surefire's `forkedProcessTimeoutInSeconds` (900 s) kills the fork, 15 min after it started, while the rerun of `testParallelIncrementWithMemStoreFlush` is still in `join()`. > > Two things that made it much worse than a plain timeout: > > 1. **Runaway flusher:** in those ~2.5 minutes the orphaned `Thread-5031` logged **7,975,132** `Flush status journal` lines (one empty flush each) and the test output hit the 1 GiB cap (`Log size exceeded the limit 1073741824, will stop logging`, 23:27:23), competing for CPU and I/O with the rerun. > 2. **Unbounded store files:** nothing compacts the standalone region, so every flush adds a file and every increment/append reads all of them. The rerun's flusher (`Thread-6616`) produced **809 store files** in ~2.5 min (405 / 302 / 102 per minute, slowing down as the file count grows). Per-method times in the surefire XML for the in-memory-flush class: `testParallelAppendWithMemStoreFlush` 356 s and `testParallelIncrementWithMemStoreFlush` 257 s, vs. 94 s and 50 s in `TestHRegion`; the class summed to ~912 s against the 780 s budget. > > So besides splitting the class, it may be worth (a) setting the done flag in a `finally` around the joins so the flusher always stops, and (b) bounding the store file count in the flusher loop (e.g. a `region.compact(true)` every N flushes), otherwise the new classes can still get close to the limit on a slow runner. Thanks for the detailed trace. The stop flags are already set in finally in this patch. I've also added compaction and compacted-file cleanup to both flushers, with a regression test covering both MemStore configurations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
