comphead commented on PR #7630: URL: https://github.com/apache/datafusion-comet/pull/7630#issuecomment-6900470240
Thanks for the careful review. Each point below is checked against the code or a run. **Benchmark.** A local A/B of the two shapes from the issue, base against this PR. The native libraries are optimized, the pool is tight, and there are three rounds with alternating order in fresh JVMs: | Query | Pool | Spills, base → PR | Spilled bytes | Warm median per round | |---|---|---|---|---| | Sort, window, sort, window, `count(DISTINCT)` | 64 MiB | sorts 7 → 4 | 21.3 → 19.3 MB | 843, 558, 561 → 668, 603, 562 ms | | Final aggregate feeding a native hash shuffle | 8 MiB | aggregate 35 → 16, writer 7 → 10 | 64.4 → 39.3 MB | 475, 455, 462 → 425, 349, 379 ms | The second row shows the cost you describe: the shuffle writer starts once the aggregate emits and spills a little more, while the total falls. For spills read back at a tight pool, `CometAggregateSuite`'s #6254 tests, where a final aggregate spills at a 3 MiB and a 2.5 MiB pool and reads its spill files back, pass with this change. So do the #6857 and #6880 PartialMerge tests. That is 769 tests across 14 `exec` suites, run locally. The A/B on the job from #7628 needs a cluster run. I'll post it here before this merges. **Ordered final aggregate.** I measured the share in use at the aggregate's first request, with the share set to the sort's own peak: | Groups per batch | Merge reservation | In use at the first request | |---|---|---| | 256 | 0 | 46% (one sorted batch), 75% (merged runs) | | 256 | 10 MiB, DataFusion's default, which Comet runs with | 2%, 3% | | 1024 | 0 | 46%, 75% | | 1024 | 10 MiB | 2%, 3% | The merge reservation makes room rather than taking it. `ExternalSorter::sort` frees it before emitting in memory (the non-spill branch of `sort()` in DataFusion 55.2). When the sort spilled, its own reservation is already zero. With 1024 groups per batch, the aggregate's first request grows from 19.5 KB to 53 KB. The test now runs all eight combinations and asserts that the sort does not spill and the aggregate completes. **DataFusion 55 failure state.** I restored `a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills`. It runs the `count(DISTINCT)` chain with the share at the lower aggregate's own peak, and with 1 KiB values so the lower aggregate keeps 97% of the share while it emits. The final aggregate spills and completes, and the lower aggregate does not spill. On `main` it spills 6 times. For DataFusion 56: apache/datafusion#25900 reserves an aggregate's initial table up front, so the resize after a spill only shrinks. A DataFusion-only test on DataFusion `main` confirmed that. **Single-reservation consumers.** Neither Comet consumer that can spill reserves only once. `ShuffleRepartitioner` reserves for every batch in `reserve_and_may_spill`, and the Iceberg fanout writer resizes as its files grow in `InnerWriter::reserve`. Both become active from their second request, and one that spilled is active again as soon as it holds memory. A consumer that reserves once and never asks again needs no share. Its bytes stay in the task's total, which every other request is checked against. `a_consumer_takes_a_share_while_it_holds_more_than_its_low_water_mark` now asserts that a sort can use what the aggregate's single reservation leaves and is refused past it. **Migration guide.** Moved under "Upgrading to Comet 1.3.0". It now says "Comet 1.3.0 divides". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
