rluvaton opened a new pull request, #25156: URL: https://github.com/apache/datafusion/pull/25156
## Which issue does this PR close? N/A ## Rationale for this change The test is flaky for non legacy with limited memory ## What changes are included in this PR? The migrated final stream replays the merged spill through an ordered table that accounts for every group of a merged batch before emitting the completed ones. With 16 groups of ~675 KiB `count(distinct)` state each and 64 row batches over 4 hash partitioned finals, that replay needed up to ~2.7 MiB, which under a `FairSpillPool` only fit once the other partitions had finished and released their share. On CI 7 consumers were still registered and the run failed with `ResourcesExhausted`. The legacy run keeps the original shape, which is the only one that reproduces the bug it guards against, and its replay stays well within its worst case share. The migrated run now uses a single final stream (no hash repartition) with 8 row batches, so it is the only spillable consumer left at replay time and a merged batch holds one or two groups. ## What is the testing strategy for this PR? N/A ## Are there any user-facing changes? <!-- If there are user-facing changes then we may require documentation to be updated before approving the PR. If there are any breaking changes to public APIs, please add the `api change` label. --> No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
