github-actions[bot] commented on code in PR #70603:
URL: https://github.com/apache/doris/pull/70603#discussion_r4581855896
##########
be/src/exec/exchange/vdata_stream_sender.cpp:
##########
@@ -288,7 +288,12 @@ Status Channel::close(RuntimeState* state) {
}
BlockSerializer::BlockSerializer(ExchangeSinkLocalState* parent, bool is_local)
- : _parent(parent), _is_local(is_local),
_batch_size(parent->state()->batch_size()) {}
+ : _parent(parent),
+ _is_local(is_local),
+ _batch_size(parent->state()->batch_size()),
+ // Adaptive scans can emit a few very wide rows per block.
Accumulating them until the
+ // row limit or EOF can exceed LZ4's input limit (e.g. 360 strings
of about 9 MB each).
+ _buffer_mem_limit(parent->state()->preferred_block_size_bytes()) {}
Review Comment:
[P2] Keep retained capacity from forcing every later batch to flush
The newly finite threshold is checked against
`_mutable_block->allocated_bytes()`, but `_serialize_block()` clears and reuses
column allocations in normal memory mode. For example, at the default 8 MiB
budget, a remote hash channel that receives a 9 MiB string retains an
allocation above the limit after sending it. Every subsequent short-string row
routed to that channel then flushes immediately, even though its current
payload is tiny. All-remote broadcast retains capacity in the same way. This
disables batching for the rest of the stream and adds serialization/compression
overhead and small packets (multi-block queuing can only coalesce blocks after
they have been serialized). Compare the current buffered payload size, or
release oversized retained capacity after a flush, and cover a wide batch
followed by multiple small batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]