peterxcli opened a new issue, #6009: URL: https://github.com/apache/datafusion-comet/issues/6009
### What is the problem the feature request solves? Since #5916 the native shuffle writer spills every output partition of a task into one file and records where each partition's blocks are in `PartitionedSpill.ranges`, a `Vec<Vec<Range<u64>>>` with one 16-byte range per partition per spill round. The lists live until the task ends: `finish_partition` reads a partition's ranges but never drops them, and no memory reservation covers them. That is 16 bytes per spilled block. With 16,000 output partitions and 11 spill rounds it is about 2.8 MB per task; a pool under pressure that spills every batch can reach 16,000 partitions × 88 rounds ≈ 22 MB per task, about 180 MB for an executor running 8 tasks, none of it visible to the memory pool. ### Describe the potential solution - Drop a partition's range list once `finish_partition` has copied it into the output. - Grow the repartitioner's `MemoryReservation` by the range size when a range is pushed and shrink it at merge. Use the infallible `grow`, so a spill cannot fail because of its own bookkeeping. - Measure retained range memory under heavy spilling (many partitions, many rounds) before and after. ### Additional context Suggested in the review of #5916: https://github.com/apache/datafusion-comet/pull/5916#issuecomment-5718670089 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
