Gabriel39 opened a new pull request, #70603:
URL: https://github.com/apache/doris/pull/70603

   ### Background
   
   External scanners adapt their row batches to the materialized output size. 
Exchange buffers combine those batches before serializing them for remote 
receivers.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: None
   PR Type: Bugfix
   
   Problem Summary:
   
   A dictionary-encoded Parquet fixture contains 360 non-null strings of 
9,247,124 bytes each. With local Exchange disabled and a large row batch, 
buffering until EOF combines 3,328,964,640 bytes of strings and exceeds LZ4's 
2,113,929,216-byte input limit. Initialize the existing Exchange buffer byte 
limit from the session's preferred block size so wide batches flush before 
accumulating to the row limit. The regression query now forces serialization 
with a large row batch instead of relying on randomized settings.
   
   ### Details
   
   - **Frequency:** Deterministic when wide input batches accumulate through a 
serialized Exchange; randomized local Exchange settings made the existing test 
intermittent.
   - **Root cause:** BlockSerializer leaves its ordinary byte threshold at 
UINT64_MAX, defeating adaptive scan batch boundaries while merging blocks.
   - **Why existing tests missed it:** The large-string fixture could bypass 
serialization through local Exchange, and existing sender tests used small 
fixed-width columns.
   - **Fix:** Reuse preferred_block_size_bytes() to initialize the existing 
byte threshold. No new configuration or wire-format change.
   - **Other paths and branches:** All buffered channels share BlockSerializer; 
existing row, EOF and low-memory flushing remain available. Direct 
serialization of an already oversized individual input block is outside this 
change.
   - **Behavior change:** Wide buffered Exchange blocks flush earlier, bounding 
accumulation before serialization.
   
   ### Testing
   
   - Reproduced the original dictionary fixture with serialized Exchange: the 
unfixed binary reports E-204 with input_size=3,328,964,640; the fixed BE 
returns 400 and 360. Switching only to local Exchange makes the unfixed control 
pass.
   - The new byte-flush unit test fails on the original implementation. After 
restoring the fix, all 21 focused Exchange unit tests pass under ASAN.
   - The standard regression runner passes the original fixture through LOCAL 
TVF with the same reader path: serialized and local Exchange each pass three 
repetitions, including the regression query hint. The complete HDFS fixture 
suite was not run locally.
   - ASAN BE build, GLIBC 2.17 compatibility audit, clang-format 16, build 
hygiene and clang-tidy pass.
   
   ### Release note
   
   Prevent Exchange buffering from combining adaptive scan batches into a 
string column larger than the serialization limit.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No.
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex.
   - Generated: Code, tests and this description.
   - Human review: Pending maintainer review.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to