comphead opened a new pull request, #7630:
URL: https://github.com/apache/datafusion-comet/pull/7630

   ## Which issue does this PR close?
   
   Closes #7628.
   
   ## Rationale for this change
   
   `CometFairMemoryPool` limits a consumer that can spill to `(task_share - 
unspillable) / num_spill`, and `num_spill` counted every registered consumer 
that can spill. DataFusion registers each operator's consumer when it builds 
the stream tree, before any batch flows (`SortExec::execute` and every 
aggregate stream's constructor in 55.2). So the operators above a blocking one 
count while they hold nothing, and the operator doing the work spills while 
most of the task's share is unused. The issue has the production numbers.
   
   A new test reproduces it on `main` with real DataFusion operators. It sorts 
the same input in the same pool, sized so the sort fits the task's share:
   
   | The sort | Spills on `main` | With this PR |
   |---|---|---|
   | Alone | 0 | 0 |
   | Below another `SortExec` | 2 | 0 |
   | Below a `Single` aggregate | 2 | 0 |
   
   ## What changes are included in this PR?
   
   - `num_spill` now counts the consumers that can spill and hold memory, plus 
the one asking. The pool updates it when a consumer's bytes go from zero to 
non-zero and back. The check of the task's total against the task's share is 
unchanged, so a consumer that starts later is refused until the others give 
memory back.
   - This reverses one decision of #7096. A sort below an 
`OrderedFinalAggregateStream`, which has no spill path over fully sorted input, 
was held to half of the share so the aggregate kept the other half. The sort 
can now fill the share while the aggregate is idle. It gives part of that back 
before the aggregate's first request: everything if it spilled, and otherwise 
what `ExternalSorter` reserves beyond its sorted runs. With the share set to 
the sort's own peak, the aggregate's first request found 47% of the share in 
use when the sort emits from one batch and 75% when it merges runs. A test pins 
this.
   - The contributor guide, the tuning guide, the migration note and the memory 
review skill describe the new divisor. Two code comments that gave "would halve 
the limit of a sort below it" as a reason are updated.
   - Existing unit tests that relied on an idle consumer taking a share now 
make that consumer hold memory, so they still test what they meant to.
   
   A pre-existing failure stays and can be reached one more way. When a 
DataFusion 55.2 hash aggregate's first request is refused, it spills and then 
resizes to what its emptied table reports. If that is more than the task has 
free, the query fails with "Decreasing allocation after spilling should 
succeed". I measured the emptied size by filling the task to leave a given 
headroom at a `Single` aggregate's first request:
   
   | Grouping key | Aggregate | Emptied table reports | Fails when free is 
below |
   |---|---|---|---|
   | `Int64` | `count`, `hll_sketch_agg` | 0 bytes | never |
   | `Int64` | `collect_list` | 24 bytes (`datatype.size()`) | 24 bytes |
   | `Utf8` | any | about 8 KiB (`GroupValuesBytes` preallocates 
`INITIAL_BUFFER_CAPACITY`) | about 8 KiB |
   
   `main` gives identical results in the same state, for example when consumers 
that cannot spill hold the task's share. With this PR, an operator below that 
can spill and filled the share while the aggregate was idle can also produce 
that state, but only if it still holds all but those last bytes when the 
aggregate starts. In the shapes I measured it gives back far more first: sorts 
at least a quarter of their peak, and the lower aggregate of a 
`count(DISTINCT)` chain at least 3% of the share even when forced. The robust 
fix is in DataFusion, which could record the emptied table with an infallible 
resize, since that memory already exists.
   
   #7096's per-task shares are not in `1.2.0` or `branch-1.2`, so this needs no 
backport.
   
   ## How are these changes tested?
   
   New tests in `fair_pool.rs` run real DataFusion plans on 
`CometFairMemoryPool` with the fake Spark. Each fails against the pool logic on 
`main`:
   
   - `the_operators_above_a_sort_take_no_share_until_they_hold_memory` is the 
table above.
   - `a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_it` 
sets the share to the sort's own peak and runs an `OrderedFinalAggregateStream` 
above it, through both the single-batch and the merge emission paths. The sort 
does not spill and the aggregate completes. On `main` the sort spills twice.
   - `a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills` 
runs Spark's `count(DISTINCT)` chain (`Single` keyed by `(k, d)`, then 
`Partial` and `Final` keyed by `k`) with the share set to the lower aggregate's 
own peak and 1 KiB values. The `Final` aggregate starts with 97% of the share 
in use, spills twice and completes, and the lower aggregate does not spill. On 
`main` the lower aggregate spills 6 times.
   
   I ran the memory pool tests, the Iceberg fanout writer share test and the 
aggregate planner tests, plus `cargo clippy --all-targets --workspace -- -D 
warnings`, `cargo fmt` and `prettier`.
   
   Not run yet: the JVM suites, of which `CometAggregateSuite`'s #6254, #6857 
and #6880 spill tests are the most relevant, the Spark SQL tests, and a 
benchmark. The memory review skill asks for benchmark evidence on a change to 
this divisor, so this stays a draft until an A/B on the job from the issue is 
done.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to