comphead opened a new pull request, #7630: URL: https://github.com/apache/datafusion-comet/pull/7630
## Which issue does this PR close? Closes #7628. ## Rationale for this change `CometFairMemoryPool` limits a consumer that can spill to `(task_share - unspillable) / num_spill`, and `num_spill` counted every registered consumer that can spill. DataFusion registers each operator's consumer when it builds the stream tree, before any batch flows (`SortExec::execute` and every aggregate stream's constructor in 55.2). So the operators above a blocking one count while they hold nothing, and the operator doing the work spills while most of the task's share is unused. The issue has the production numbers. A new test reproduces it on `main` with real DataFusion operators. It sorts the same input in the same pool, sized so the sort fits the task's share: | The sort | Spills on `main` | With this PR | |---|---|---| | Alone | 0 | 0 | | Below another `SortExec` | 2 | 0 | | Below a `Single` aggregate | 2 | 0 | ## What changes are included in this PR? - `num_spill` now counts the consumers that can spill and hold memory, plus the one asking. The pool updates it when a consumer's bytes go from zero to non-zero and back. The check of the task's total against the task's share is unchanged, so a consumer that starts later is refused until the others give memory back. - This reverses one decision of #7096. A sort below an `OrderedFinalAggregateStream`, which has no spill path over fully sorted input, was held to half of the share so the aggregate kept the other half. The sort can now fill the share while the aggregate is idle. It gives part of that back before the aggregate's first request: everything if it spilled, and otherwise what `ExternalSorter` reserves beyond its sorted runs. With the share set to the sort's own peak, the aggregate's first request found 47% of the share in use when the sort emits from one batch and 75% when it merges runs. A test pins this. - The contributor guide, the tuning guide, the migration note and the memory review skill describe the new divisor. Two code comments that gave "would halve the limit of a sort below it" as a reason are updated. - Existing unit tests that relied on an idle consumer taking a share now make that consumer hold memory, so they still test what they meant to. A pre-existing failure stays and can be reached one more way. When a DataFusion 55.2 hash aggregate's first request is refused, it spills and then resizes to what its emptied table reports. If that is more than the task has free, the query fails with "Decreasing allocation after spilling should succeed". I measured the emptied size by filling the task to leave a given headroom at a `Single` aggregate's first request: | Grouping key | Aggregate | Emptied table reports | Fails when free is below | |---|---|---|---| | `Int64` | `count`, `hll_sketch_agg` | 0 bytes | never | | `Int64` | `collect_list` | 24 bytes (`datatype.size()`) | 24 bytes | | `Utf8` | any | about 8 KiB (`GroupValuesBytes` preallocates `INITIAL_BUFFER_CAPACITY`) | about 8 KiB | `main` gives identical results in the same state, for example when consumers that cannot spill hold the task's share. With this PR, an operator below that can spill and filled the share while the aggregate was idle can also produce that state, but only if it still holds all but those last bytes when the aggregate starts. In the shapes I measured it gives back far more first: sorts at least a quarter of their peak, and the lower aggregate of a `count(DISTINCT)` chain at least 3% of the share even when forced. The robust fix is in DataFusion, which could record the emptied table with an infallible resize, since that memory already exists. #7096's per-task shares are not in `1.2.0` or `branch-1.2`, so this needs no backport. ## How are these changes tested? New tests in `fair_pool.rs` run real DataFusion plans on `CometFairMemoryPool` with the fake Spark. Each fails against the pool logic on `main`: - `the_operators_above_a_sort_take_no_share_until_they_hold_memory` is the table above. - `a_sort_that_fills_the_share_leaves_room_for_the_final_aggregate_above_it` sets the share to the sort's own peak and runs an `OrderedFinalAggregateStream` above it, through both the single-batch and the merge emission paths. The sort does not spill and the aggregate completes. On `main` the sort spills twice. - `a_final_aggregate_that_starts_while_the_one_below_fills_the_share_spills` runs Spark's `count(DISTINCT)` chain (`Single` keyed by `(k, d)`, then `Partial` and `Final` keyed by `k`) with the share set to the lower aggregate's own peak and 1 KiB values. The `Final` aggregate starts with 97% of the share in use, spills twice and completes, and the lower aggregate does not spill. On `main` the lower aggregate spills 6 times. I ran the memory pool tests, the Iceberg fanout writer share test and the aggregate planner tests, plus `cargo clippy --all-targets --workspace -- -D warnings`, `cargo fmt` and `prettier`. Not run yet: the JVM suites, of which `CometAggregateSuite`'s #6254, #6857 and #6880 spill tests are the most relevant, the Spark SQL tests, and a benchmark. The memory review skill asks for benchmark evidence on a change to this divisor, so this stays a draft until an A/B on the job from the issue is done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
