cetra3 commented on code in PR #20642:
URL: https://github.com/apache/datafusion/pull/20642#discussion_r2916178941
##########
datafusion/physical-plan/src/sorts/multi_level_merge.rs:
##########
@@ -356,45 +363,57 @@ impl MultiLevelMergeBuilder {
) -> Result<(Vec<SortedSpillFile>, usize)> {
assert_ne!(buffer_len, 0, "Buffer length must be greater than 0");
let mut number_of_spills_to_read_for_current_phase = 0;
+ // Track total memory needed for spill file buffers. When the
+ // reservation has pre-reserved bytes (from
sort_spill_reservation_bytes),
+ // those bytes cover the first N spill files without additional pool
+ // allocation, preventing starvation under memory pressure.
+ let mut total_needed: usize = 0;
for spill in &self.sorted_spill_files {
- // For memory pools that are not shared this is good, for other
this is not
- // and there should be some upper limit to memory reservation so
we won't starve the system
Review Comment:
I think this comment still applies: if you have multiple partitions running,
one partition will still be able to starve the others
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
