github-actions[bot] commented on code in PR #70603:
URL: https://github.com/apache/doris/pull/70603#discussion_r4581855896


##########
be/src/exec/exchange/vdata_stream_sender.cpp:
##########
@@ -288,7 +288,12 @@ Status Channel::close(RuntimeState* state) {
 }
 
 BlockSerializer::BlockSerializer(ExchangeSinkLocalState* parent, bool is_local)
-        : _parent(parent), _is_local(is_local), 
_batch_size(parent->state()->batch_size()) {}
+        : _parent(parent),
+          _is_local(is_local),
+          _batch_size(parent->state()->batch_size()),
+          // Adaptive scans can emit a few very wide rows per block. 
Accumulating them until the
+          // row limit or EOF can exceed LZ4's input limit (e.g. 360 strings 
of about 9 MB each).
+          _buffer_mem_limit(parent->state()->preferred_block_size_bytes()) {}

Review Comment:
   [P2] Keep retained capacity from forcing every later batch to flush
   
   The newly finite threshold is checked against 
`_mutable_block->allocated_bytes()`, but `_serialize_block()` clears and reuses 
column allocations in normal memory mode. For example, at the default 8 MiB 
budget, a remote hash channel that receives a 9 MiB string retains an 
allocation above the limit after sending it. Every subsequent short-string row 
routed to that channel then flushes immediately, even though its current 
payload is tiny. All-remote broadcast retains capacity in the same way. This 
disables batching for the rest of the stream and adds serialization/compression 
overhead and small packets (multi-block queuing can only coalesce blocks after 
they have been serialized). Compare the current buffered payload size, or 
release oversized retained capacity after a flush, and cover a wide batch 
followed by multiple small batches.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to