dwsmith1983 commented on code in PR #5565:
URL: https://github.com/apache/datafusion-comet/pull/5565#discussion_r3891765020
##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -64,6 +65,9 @@ enum DataOutput {
pub(crate) struct LocalPartitionWriter {
output_index_file: String,
data_output: DataOutput,
+ /// Compression state shared by every block this task writes; the
per-partition
+ /// `BufBatchWriter`s borrow it (see [`ShuffleCodecContext`]).
+ codec_context: ShuffleCodecContext,
Review Comment:
Same treatment here, plus boundary releases: the writer drops its context
when each spill event completes and after the final flush, and any block that
leaves the context above 8 MiB drops it immediately. So retained memory between
phases is zero and the worst case anywhere is the cap, matching the base path
profile rather than adding a new reservation surface. The size-based regression
drives a real repartitioner through spill and finish and checks the context is
gone at both points; level 22 locally measures ~834 MB retained without the
cap, which lines up with your numbers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
