comphead commented on PR #7630:
URL: 
https://github.com/apache/datafusion-comet/pull/7630#issuecomment-6900470240

   Thanks for the careful review. Each point below is checked against the code 
or a run.
   
   **Benchmark.** A local A/B of the two shapes from the issue, base against 
this PR. The native libraries are optimized, the pool is tight, and there are 
three rounds with alternating order in fresh JVMs:
   
   | Query | Pool | Spills, base → PR | Spilled bytes | Warm median per round |
   |---|---|---|---|---|
   | Sort, window, sort, window, `count(DISTINCT)` | 64 MiB | sorts 7 → 4 | 
21.3 → 19.3 MB | 843, 558, 561 → 668, 603, 562 ms |
   | Final aggregate feeding a native hash shuffle | 8 MiB | aggregate 35 → 16, 
writer 7 → 10 | 64.4 → 39.3 MB | 475, 455, 462 → 425, 349, 379 ms |
   
   The second row shows the cost you describe: the shuffle writer starts once 
the aggregate emits and spills a little more, while the total falls. For spills 
read back at a tight pool, `CometAggregateSuite`'s #6254 tests, where a final 
aggregate spills at a 3 MiB and a 2.5 MiB pool and reads its spill files back, 
pass with this change. So do the #6857 and #6880 PartialMerge tests. That is 
769 tests across 14 `exec` suites, run locally. The A/B on the job from #7628 
needs a cluster run. I'll post it here before this merges.
   
   **Ordered final aggregate.** I measured the share in use at the aggregate's 
first request, with the share set to the sort's own peak:
   
   | Groups per batch | Merge reservation | In use at the first request |
   |---|---|---|
   | 256 | 0 | 46% (one sorted batch), 75% (merged runs) |
   | 256 | 10 MiB, DataFusion's default, which Comet runs with | 2%, 3% |
   | 1024 | 0 | 46%, 75% |
   | 1024 | 10 MiB | 2%, 3% |
   
   The merge reservation makes room rather than taking it. 
`ExternalSorter::sort` frees it before emitting in memory (the non-spill branch 
of `sort()` in DataFusion 55.2). When the sort spilled, its own reservation is 
already zero. With 1024 groups per batch, the aggregate's first request grows 
from 19.5 KB to 53 KB. The test now runs all eight combinations and asserts 
that the sort does not spill and the aggregate completes.
   
   **DataFusion 55 failure state.** I restored 
`a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills`. It 
runs the `count(DISTINCT)` chain with the share at the lower aggregate's own 
peak, and with 1 KiB values so the lower aggregate keeps 97% of the share while 
it emits. The final aggregate spills and completes, and the lower aggregate 
does not spill. On `main` it spills 6 times. For DataFusion 56: 
apache/datafusion#25900 reserves an aggregate's initial table up front, so the 
resize after a spill only shrinks. A DataFusion-only test on DataFusion `main` 
confirmed that.
   
   **Single-reservation consumers.** Neither Comet consumer that can spill 
reserves only once. `ShuffleRepartitioner` reserves for every batch in 
`reserve_and_may_spill`, and the Iceberg fanout writer resizes as its files 
grow in `InnerWriter::reserve`. Both become active from their second request, 
and one that spilled is active again as soon as it holds memory. A consumer 
that reserves once and never asks again needs no share. Its bytes stay in the 
task's total, which every other request is checked against. 
`a_consumer_takes_a_share_while_it_holds_more_than_its_low_water_mark` now 
asserts that a sort can use what the aggregate's single reservation leaves and 
is refused past it.
   
   **Migration guide.** Moved under "Upgrading to Comet 1.3.0". It now says 
"Comet 1.3.0 divides".
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to