comphead opened a new issue, #7628:
URL: https://github.com/apache/datafusion-comet/issues/7628

   ### Describe the bug
   
   `CometFairMemoryPool` limits a consumer that can spill to `(task_share - 
unspillable) / num_spill`, where `num_spill` counts every registered consumer 
that can spill 
([fair_pool.rs](https://github.com/apache/datafusion-comet/blob/main/native/core/src/execution/memory_pools/fair_pool.rs)).
 DataFusion registers consumers when `execute()` builds the stream tree, before 
any batch flows. So the consumers above a blocking operator count from the 
start, although they hold nothing until it emits. The operator doing the work 
is capped at a fraction of the task's share and spills while most of that share 
is unused.
   
   This is the per-task form of apache/datafusion#22036, which describes the 
same policy in `FairSpillPool`. Since #7088 the per-consumer limit binds 
whenever tasks run concurrently, so it now shows up in ordinary plans.
   
   Two stages of one Spark 4.0 job, on Comet main with DataFusion main. Each 
executor has 8 cores and `spark.memory.offHeap.size=24g`, so a task's share is 
3 GiB with 8 tasks running. Spill compression is zstd.
   
   **Sort, Window, Sort, Window, then four HashAggregates** (the plan of a 
`count(DISTINCT)` after two window functions). 216 tasks each sort about 4.95M 
rows, 1.08 GB of Arrow data:
   
   | | Comet | Spark |
   |---|---|---|
   | Spills | 1,080 and 860 for the two sorts, 5 and 4 per task | only when the 
executor's pool was full |
   | Spilled to disk | 56 GB | 19.7 GB |
   | Peak off-heap execution memory per task | 653 MiB | 3 GiB |
   | Free off-heap per executor during the stage, median / min | 21.8 / 19.0 
GiB | 10.3 / 0.0 GiB |
   
   The two sorts and the four aggregates all register as consumers that can 
spill, so each sort is held to about 3 GiB / 6 = 0.5 GiB. `ExternalSorter` also 
reserves about twice the bytes it buffers 
(`get_reserved_bytes_for_record_batch`). At their peaks the pools' reservations 
were 1.7x to 1.9x the live native allocations. So each sorted run holds about 
0.25 GB.
   
   **Final HashAggregate with 17 `hll_sketch_agg` columns, feeding a RoundRobin 
native shuffle writer**, 216 tasks. The aggregate and `ShuffleRepartitioner` 
get 1.5 GiB each. The shuffle writer receives nothing until the aggregate 
emits, so its half stays unused.
   
   | Tasks | Peak off-heap execution memory | Spilled | Task time |
   |---|---|---|---|
   | 209 tasks on executors running only this stage | 1.5 GiB | yes (818 
spills, 8.1 GB in total) | 48 to 56 s |
   | 7 tasks that shared an executor with tasks of a non-native stage, so had a 
larger share | 5.1 GiB | no | 30 to 31 s |
   
   Free off-heap per executor while those tasks spilled: median 19 to 20 GiB, 
minimum 12 GiB.
   
   ### Steps to reproduce
   
   A task plan where a blocking operator that can spill sits below other 
operators that can spill, run with enough concurrent tasks that `off_heap_size 
/ running_tasks` binds. Examples are a sort below a window and an aggregate, or 
a final aggregate feeding a native shuffle writer. I do not have a minimal 
reproducer yet.
   
   ### Expected behavior
   
   A consumer doing the work can use the memory that registered but idle 
consumers are not using, as long as the task stays within its share.
   
   ### Additional context
   
   Possible directions:
   
   - Count only the consumers that hold memory, or have reserved any, in 
`num_spill`, and recompute the limit when another one starts.
   - Let a consumer exceed its fair limit while the task's total stays within 
`task_share`, and take memory back by asking it to spill when another consumer 
starts. This needs a reclaim path, see #3873 and the closed 
apache/datafusion#22043.
   
   Today `spark.comet.exec.memoryPool=greedy_unified` avoids the split, with 
the risks that #7088 and #7197 describe.
   
   Related: apache/datafusion#22036, #7088, #7197, #3873, #7472.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to