XiaoHongbo-Hope opened a new pull request, #9209: URL: https://github.com/apache/paimon/pull/9209
### Purpose Fix Data Evolution reads failing when ordinary variable-length columns exceed the Arrow 2 GiB offset limit across input batches: ```text pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays ``` `MergeAllBatchReader` previously collected every supplier batch and forced each column into one Arrow array. This overflows the 32-bit offsets used by `string`, `binary`, and similar types. ### Changes - Stream supplier batches instead of materializing and concatenating the complete range. - Bound returned batches with zero-copy slices. - Preserve buffered remainders in `DataEvolutionMergeReader` without concatenating them with the next batch. - Keep the original Arrow schema; columns are not changed to `large_string` or `large_binary`. ### Tests The regression tests inject a failure into `pa.concat_arrays`, reproducing the old failing path without allocating more than 2 GiB, and verify both supplier streaming and remainder alignment. - Data Evolution and row-id suites: 100 passed, 1 skipped - BLOB-related suite: 141 passed, 1 skipped - Python 3.6 + PyArrow 6 focused suite: 14 passed - flake8, compileall, and `git diff --check` passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
