sunchao commented on code in PR #5568:
URL: https://github.com/apache/datafusion-comet/pull/5568#discussion_r3897155713
##########
native/shuffle/src/writers/local/local_partition_writer.rs:
##########
@@ -53,6 +53,11 @@ enum DataOutput {
spill_writers: Vec<SpillWriter>,
/// Runtime used to allocate the temporary spill files.
runtime: Arc<RuntimeEnv>,
+ /// Byte buffer recycled through the short-lived per-partition
`BufBatchWriter`s.
+ /// Partitions are written strictly one at a time, so a single buffer
keeps its
+ /// grown capacity across the whole task instead of every partition
regrowing a
+ /// fresh allocation toward the write buffer size.
+ recycled_buffer: Vec<u8>,
Review Comment:
[P2] Could you extend the requested benchmark with a wide-then-narrow
forced-spill case to quantify this retained-buffer tradeoff? At `3b4425cb`, a
successful spill returns the byte buffer to `LocalPartitionWriter` before
freeing the reservation, and the write threshold is checked only after a whole
block is encoded. This establishes a lifetime/accounting change, not a measured
RSS regression.
One bounded case would use an early hot-key batch of 8,192 rows with 8-KiB
Binary values, followed by small values with the same schema. Compare that with
reversed-order and narrow-only controls, using `None` compression and a
forced-spill threshold. Compare base `199a910bd8623cac3db82d70f9bfb24865613c2c`
against head `3b4425cb4b0c7e0688248f6c2571c08307c3fdb1` with the same release
build, allocator, hardware, input seed, warmup and repeated runs.
Could you report total time, allocation/growth bytes, retained buffer
capacity and tracked reservation after each spill, peak/live memory, spill
bytes, and matching decoded rows/order? This would complement the existing
high-partition throughput request and help choose whether to cap, release or
account for the retained buffer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
