Phoenix500526 opened a new issue, #25011: URL: https://github.com/apache/datafusion/issues/25011
### Describe the bug The `cargo test hash collisions (amd64)` job can spend several hours in two count-distinct unit tests and reach the 360-minute execution limit. I noticed this while checking #24923, then found the same behavior on `main` and other branches. The tests are in `datafusion-functions-aggregate-common`: ```text aggregate::count_distinct::bytes::tests::ungrouped_utf8_accumulator_is_never_worse_than_a_pre_allocated_set aggregate::count_distinct::bytes::tests::ungrouped_utf8_view_accumulator_is_never_worse_than_a_pre_allocated_set ``` Examples from September 6-7, 2026: - [main at d25ffaab493, job 101570989117](https://github.com/apache/datafusion/actions/runs/34064580597/job/101570989117): the test executable started at 22:42:59 UTC. The Utf8 test completed at 02:40:34, almost four hours later. The Utf8View test had no completion entry before the job was canceled at 04:38 UTC. - [Another branch, job 101586763994](https://github.com/apache/datafusion/actions/runs/34070465339/job/101586763994): the executable started at 00:49:29 UTC. Utf8 completed at 06:11:56, and Utf8View had no completion entry before cancellation at 06:39 UTC. - [A third job, 101585196441](https://github.com/apache/datafusion/actions/runs/34069903467/job/101585196441), also reached the six-hour limit. All three jobs have the annotation: `The job has exceeded the maximum execution time of 6h0m0s`. These are timeout cancellations, not skipped tests. It does not always time out: [main at fdfb67d056](https://github.com/apache/datafusion/actions/runs/34071340862/job/101590431776) completed successfully. The Utf8 and Utf8View tests took approximately 2h39m and 4h31m respectively, and the crate reported `49 passed; 0 failed; ... finished in 16248.36s`. ### To Reproduce The tests were added in #24857, merged as `d25ffaab493c18690028410656ed12ea6c6fb0b6`. On a checkout containing that commit, compare: ```bash # Normal hashing cargo test --profile ci -p datafusion-functions-aggregate-common \ --lib ungrouped_utf8 -- --nocapture # Forced collisions, as enabled in the affected CI job. # This can run for hours; interrupt after collecting enough evidence. cargo test --profile ci -p datafusion-functions-aggregate-common \ --features datafusion-common/force_hash_collisions \ --lib ungrouped_utf8 -- --nocapture ``` ### Expected behavior The collision check should finish within the CI time limit while preserving the regression coverage these tests were designed to provide. Is this already tracked, or is there a plan for handling these tests in the forced-collision job? In particular, what behavior or regression boundaries are the 100,000 and 500,000 cardinalities intended to protect, and what approach would preserve that coverage? ### Additional context The [test cardinalities](https://github.com/apache/datafusion/blob/d25ffaab493c18690028410656ed12ea6c6fb0b6/datafusion/functions-aggregate-common/src/aggregate/count_distinct/bytes.rs#L180-L197) reach 500,000 distinct strings, inserted into both a lazy and a pre-allocated set. The comments describe checking allocation savings and capacity convergence. With [`force_hash_collisions`](https://github.com/apache/datafusion/blob/d25ffaab493c18690028410656ed12ea6c6fb0b6/datafusion/common/src/hash_utils.rs#L1184-L1195), every hash is zero. Source inspection suggests quadratic collision checking explains the long runtime. #24918 describes a related forced-collision problem, but fixes a different memory-limit test whose partitioning assumptions no longer hold. It does not address these two tests. I would like to understand the intended coverage and the community's preferred approach before proposing any reduction in cardinality or skipping tests. This issue is reporting the CI impact, not proposing either change as the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
