rluvaton commented on code in PR #20642:
URL: https://github.com/apache/datafusion/pull/20642#discussion_r2878879152
##########
datafusion/physical-plan/src/sorts/sort.rs:
##########
@@ -355,6 +355,13 @@ impl ExternalSorter {
self.sort_and_spill_in_mem_batches().await?;
}
+ // Transfer the pre-reserved merge memory to the streaming merge
+ // using `take()` instead of `new_empty()`. This ensures the merge
+ // stream starts with `sort_spill_reservation_bytes` already
+ // allocated, preventing starvation when concurrent sort partitions
+ // compete for pool memory. `take()` moves the bytes atomically
+ // without releasing them back to the pool, so other partitions
+ // cannot race to consume the freed memory.
Review Comment:
The pre reserved merge memory should be used as part of the sort merge
stream.
I mean that if x pre reserved merge memory was reserved the sort merge
stream should know about that so it wont think it starting from 0, otherwise
this just reserve for unaccounted memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
