kumarUjjawal opened a new issue, #27033: URL: https://github.com/apache/datafusion/issues/27033
### Is your feature request related to a problem or challenge? `aggregates::clustered_partial_stream::tests::completed_groups_release_first_value_bitmap` (added in #26782) takes about 10 minutes in the `cargo test hash collisions (amd64)` merge-queue job. Without `force_hash_collisions`, it takes about 1 second. The test aggregates 131,072 distinct `Int64` keys in one batch. Under `force_hash_collisions`, `GroupValuesPrimitive` gives every key the hash 0, so each insert compares against all earlier keys. The test also runs the plan two times (without a limit and with a 16 KB limit). The test checks that `FIRST_VALUE(v ORDER BY v)` releases its per-batch bitmap. Colliding hashes do not change what it checks. Evidence from merge-queue runs: - In the passing rerun for #26926 ([job](https://github.com/apache/datafusion/actions/runs/41571721394/job/125710730373)), the physical-plan unit test binary took 609.89s. All other tests in the binary finished by 03:40:31. This test ran from about 03:40:07 to 03:50:17. - The median time of the job went from 6.2 minutes (230 jobs from Oct 21 until #26782 merged on Oct 30) to 12.0 minutes (70 jobs after). - All 5 failed attempts of the job since #26782 merged were spot runner shutdowns ("The runner has received a shutdown signal") while this test was running. Each one failed the merge-queue run for that PR, and a rerun passed: - #26929: [job](https://github.com/apache/datafusion/actions/runs/41114163010/job/124263887540) - #26898: [job](https://github.com/apache/datafusion/actions/runs/41115099549/job/124267135386) - #26883: [job](https://github.com/apache/datafusion/actions/runs/41120437208/job/124285496814) - #26974: [job](https://github.com/apache/datafusion/actions/runs/41559552415/job/125666576696) - #26926: [job](https://github.com/apache/datafusion/actions/runs/41571721394/job/125705896210) Locally (`--profile ci`, single test), it takes 0.92s by default and 309s with `force_hash_collisions`. ### Describe the solution you'd like Skip the test under `force_hash_collisions`, as #26594 did for `sort_after_group_by_reserves_shared_output_once`. The regular `cargo test` job still runs it. ### Describe alternatives you've considered Use fewer groups with a lower memory limit. This keeps the test in the hash-collision job, but the bitmap must still be larger than what the active group needs, so it is a call for the test's author. ### Additional context Related to #26759. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
