Gabriel39 opened a new pull request, #70603: URL: https://github.com/apache/doris/pull/70603
### Background External scanners adapt their row batches to the materialized output size. Exchange buffers combine those batches before serializing them for remote receivers. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Bugfix Problem Summary: A dictionary-encoded Parquet fixture contains 360 non-null strings of 9,247,124 bytes each. With local Exchange disabled and a large row batch, buffering until EOF combines 3,328,964,640 bytes of strings and exceeds LZ4's 2,113,929,216-byte input limit. Initialize the existing Exchange buffer byte limit from the session's preferred block size so wide batches flush before accumulating to the row limit. The regression query now forces serialization with a large row batch instead of relying on randomized settings. ### Details - **Frequency:** Deterministic when wide input batches accumulate through a serialized Exchange; randomized local Exchange settings made the existing test intermittent. - **Root cause:** BlockSerializer leaves its ordinary byte threshold at UINT64_MAX, defeating adaptive scan batch boundaries while merging blocks. - **Why existing tests missed it:** The large-string fixture could bypass serialization through local Exchange, and existing sender tests used small fixed-width columns. - **Fix:** Reuse preferred_block_size_bytes() to initialize the existing byte threshold. No new configuration or wire-format change. - **Other paths and branches:** All buffered channels share BlockSerializer; existing row, EOF and low-memory flushing remain available. Direct serialization of an already oversized individual input block is outside this change. - **Behavior change:** Wide buffered Exchange blocks flush earlier, bounding accumulation before serialization. ### Testing - Reproduced the original dictionary fixture with serialized Exchange: the unfixed binary reports E-204 with input_size=3,328,964,640; the fixed BE returns 400 and 360. Switching only to local Exchange makes the unfixed control pass. - The new byte-flush unit test fails on the original implementation. After restoring the fix, all 21 focused Exchange unit tests pass under ASAN. - The standard regression runner passes the original fixture through LOCAL TVF with the same reader path: serialized and local Exchange each pass three repetitions, including the regression query hint. The complete HDFS fixture suite was not run locally. - ASAN BE build, GLIBC 2.17 compatibility audit, clang-format 16, build hygiene and clang-tidy pass. ### Release note Prevent Exchange buffering from combining adaptive scan batches into a string column larger than the serialization limit. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Ready ### AI Disclosure - Tool: Codex. - Generated: Code, tests and this description. - Human review: Pending maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
