comphead opened a new issue, #7628: URL: https://github.com/apache/datafusion-comet/issues/7628
### Describe the bug `CometFairMemoryPool` limits a consumer that can spill to `(task_share - unspillable) / num_spill`, where `num_spill` counts every registered consumer that can spill ([fair_pool.rs](https://github.com/apache/datafusion-comet/blob/main/native/core/src/execution/memory_pools/fair_pool.rs)). DataFusion registers consumers when `execute()` builds the stream tree, before any batch flows. So the consumers above a blocking operator count from the start, although they hold nothing until it emits. The operator doing the work is capped at a fraction of the task's share and spills while most of that share is unused. This is the per-task form of apache/datafusion#22036, which describes the same policy in `FairSpillPool`. Since #7088 the per-consumer limit binds whenever tasks run concurrently, so it now shows up in ordinary plans. Two stages of one Spark 4.0 job, on Comet main with DataFusion main. Each executor has 8 cores and `spark.memory.offHeap.size=24g`, so a task's share is 3 GiB with 8 tasks running. Spill compression is zstd. **Sort, Window, Sort, Window, then four HashAggregates** (the plan of a `count(DISTINCT)` after two window functions). 216 tasks each sort about 4.95M rows, 1.08 GB of Arrow data: | | Comet | Spark | |---|---|---| | Spills | 1,080 and 860 for the two sorts, 5 and 4 per task | only when the executor's pool was full | | Spilled to disk | 56 GB | 19.7 GB | | Peak off-heap execution memory per task | 653 MiB | 3 GiB | | Free off-heap per executor during the stage, median / min | 21.8 / 19.0 GiB | 10.3 / 0.0 GiB | The two sorts and the four aggregates all register as consumers that can spill, so each sort is held to about 3 GiB / 6 = 0.5 GiB. `ExternalSorter` also reserves about twice the bytes it buffers (`get_reserved_bytes_for_record_batch`). At their peaks the pools' reservations were 1.7x to 1.9x the live native allocations. So each sorted run holds about 0.25 GB. **Final HashAggregate with 17 `hll_sketch_agg` columns, feeding a RoundRobin native shuffle writer**, 216 tasks. The aggregate and `ShuffleRepartitioner` get 1.5 GiB each. The shuffle writer receives nothing until the aggregate emits, so its half stays unused. | Tasks | Peak off-heap execution memory | Spilled | Task time | |---|---|---|---| | 209 tasks on executors running only this stage | 1.5 GiB | yes (818 spills, 8.1 GB in total) | 48 to 56 s | | 7 tasks that shared an executor with tasks of a non-native stage, so had a larger share | 5.1 GiB | no | 30 to 31 s | Free off-heap per executor while those tasks spilled: median 19 to 20 GiB, minimum 12 GiB. ### Steps to reproduce A task plan where a blocking operator that can spill sits below other operators that can spill, run with enough concurrent tasks that `off_heap_size / running_tasks` binds. Examples are a sort below a window and an aggregate, or a final aggregate feeding a native shuffle writer. I do not have a minimal reproducer yet. ### Expected behavior A consumer doing the work can use the memory that registered but idle consumers are not using, as long as the task stays within its share. ### Additional context Possible directions: - Count only the consumers that hold memory, or have reserved any, in `num_spill`, and recompute the limit when another one starts. - Let a consumer exceed its fair limit while the task's total stays within `task_share`, and take memory back by asking it to spill when another consumer starts. This needs a reclaim path, see #3873 and the closed apache/datafusion#22043. Today `spark.comet.exec.memoryPool=greedy_unified` avoids the split, with the risks that #7088 and #7197 describe. Related: apache/datafusion#22036, #7088, #7197, #3873, #7472. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
