andygrove commented on issue #6253: URL: https://github.com/apache/datafusion-comet/issues/6253#issuecomment-7017135720
I'm closing #6279 without merging. This records what it did, what it measured, and what is left, so the work can be picked up again. The branch is kept at `andygrove/datafusion-comet:fix-6253-window-memory` (head `b87b0dc964`, mergeable against main when it was closed). ## The gap DataFusion's window operators register no `MemoryConsumer`. `WindowAggExec`, which Comet uses for `PERCENT_RANK`, `CUME_DIST`, `NTILE` and any aggregate whose frame ends at `UNBOUNDED FOLLOWING` (including Spark's default frame for `agg(x) OVER (PARTITION BY k)`), buffers the task's whole input and concatenates it without telling the pool. DataFusion 56 (`branch-56` at `964f925`) is unchanged here: nothing under `datafusion/physical-plan/src/windows/` reserves memory or spills. Upstream spilling is apache/datafusion#22946, still open. ## What #6279 did - A new `CometWindowAggExec` (`native/core/src/execution/operators/window_agg.rs`, wired in at `planner/window.rs`) that evaluates the same `WindowExpr`s one window partition at a time. It emits each window partition as soon as the sorted input moves past it and keeps only the one that may continue. Without `PARTITION BY` it still buffers everything. - It reserves what it buffers with `RecordBatchMemoryCounter` (shared buffers counted once) and reserves the concatenated copy by counting only the buffers the copy does not share with its inputs. Per-batch `get_slice_memory_size` estimates over-charged batches sliced from one list array about tenfold. - It registers with the pool when it buffers its first batch, not in `execute()`. Registering earlier shrinks the sort's share under `fair_unified` while the sort is still reading. - A refused reservation says the operator cannot spill, that a window partition needs up to twice its size within the operator's share, and which settings to change (`spark.memory.offHeap.size`, then `greedy_unified`, then `spark.comet.exec.window.enabled=false`). Reservation alone was not enough. A build that only reserved the buffered batches failed `SUM(v) OVER (PARTITION BY k)` over 200,000 rows in 5-row window partitions under a 4 MB pool, because `WindowAggExec` holds the whole task input, not one window partition. The streaming emission is what makes the reservation usable. ## Measurements (standalone bench, release, 2M rows in 8192-row batches) - Within 5% of `WindowAggExec` for 10-row and 100,000-row window partitions and with no `PARTITION BY`. 6-15% slower with 1,000-row window partitions (about 0.5 ms per 2M rows), because it concatenates per batch. - 20M rows (480 MB) in 1,000-row window partitions: `WindowAggExec` peaked at 1,254 MB RSS, reserved 0 and emitted one 20M-row batch. `CometWindowAggExec` peaked at 480 MB (the input itself), reserved 609 KB at peak and emitted 9,000-row batches. - Without `PARTITION BY` both peak at 1,090 MB, and `CometWindowAggExec` reserves 960 MB of it. ## Tests worth reusing - A Rust differential test against `WindowAggExec` over 20 seeds (random window partition sizes, random batch boundaries including empty batches, null keys, two key columns, with and without `PARTITION BY`), plus mutation-checked memory tests: no batch reservation, no copy reservation, no release after emitting, early registration, never continuing a window partition across batches, not merging a continued window partition. - `windows/window_whole_partition_batches.sql` (whole-partition functions over 7-row batches). - Two `CometWindowExecSuite` tests: many small window partitions under a 4 MB pool pass and match Spark, and a window partition that does not fit fails the task with the `CometWindowAggExec cannot spill` hint. - Spark 4.1.3 `DataFrameWindowFunctionsSuite` and `DataFrameWindowFramesSuite` gave the same result on main and on the branch (74 pass, the same 4 fail, all marked `IgnoreComet`). ## What was not done - The `fair_unified` TPC-DS window query with a pool small enough to make the sort spill, compared against main. It was asked for in review. The standalone bench does not cover whether the share dilution or the per-batch pool calls cost anything there. - `BoundedWindowAggExec` reserves nothing and is not covered. A `RANGE ... CURRENT ROW` frame keeps a whole group of peer rows, so a skewed `ORDER BY` value can still grow untracked. That is #7092. - Spilling. A window partition that exceeds the operator's share fails the task where Spark would spill. In on-heap mode the pool is unbounded, so nothing is refused there. ## Open question for whoever resumes this Comet carries its own copy of `WindowAggStream` and about ten delegated methods through every DataFusion upgrade. The alternatives are to offer the streaming-emission stream upstream as the no-spill first step of apache/datafusion#22946, or to wait for that issue to get a design. Two earlier upstream attempts (apache/datafusion#22947 and apache/datafusion#23207) were closed unmerged, and apache/datafusion#23207 only reserved while still buffering the whole input. As of 2026-11-17 I had not found an upstream design that emits completed window partitions early. Review feedback on #6279 suggested merging the DataFusion 56 upgrade first and then reanalyzing what is still open here. Related DataFusion window work filed by comphead: apache/datafusion#25637 (spilling coverage epic), and apache/datafusion#26537, apache/datafusion#26539 and apache/datafusion#26541 (`BoundedWindowAggExec` buffering and per-partition cost). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
