discivigour opened a new pull request, #9146: URL: https://github.com/apache/paimon/pull/9146
## What changed - Preserve Arrow columns as `ChunkedArray` during update-by-row-id instead of flattening the entire column with `combine_chunks()`. - Merge updates within each original chunk and recursively split a row range when `replace_with_mask`, `concat_arrays`, `take`, or type coercion reaches a 32-bit Arrow offset limit. - Select replacement values from their source chunks without asking Arrow to concatenate unrelated chunks. - Add low-memory regression coverage for VARIANT structs, columns whose aggregate offsets exceed `int32`, fallback concatenation overflow, and `large_list` to `list` coercion. ## Why Arrow `binary`, `string`, and `list` arrays use signed 32-bit offsets. A table column can legally exceed 2 GiB by retaining multiple chunks, but `TableUpdateByRowId._merge_update_with_original` previously called `combine_chunks()` on the complete original and update columns. This failed with `ArrowInvalid: offset overflow` for large data-evolution files. VARIANT columns also use a struct containing binary children. Because `replace_with_mask` has no struct/list kernel, the fallback temporarily concatenated the original and replacement arrays and could hit the same limit even when the final result was representable. The new path keeps valid chunks independent and only subdivides a chunk when a local intermediate result would overflow. It preserves the original Arrow schema, including VARIANT child nullability, and the existing writer already consumes the resulting table batch by batch. ## Validation - `python -m unittest pypaimon.tests.table_update_by_row_id_chunked_test` with PyArrow 19.0.1: 4 tests passed. - The same regression test with Python 3.6 / PyArrow 6.0.1: 4 tests passed. - `python -m unittest pypaimon.tests.table_update_by_row_id_chunked_test pypaimon.tests.table_update_test` with PyArrow 16.0.0: 100 tests passed. - `flake8 --config=dev/cfg.ini pypaimon/write/table_update_by_row_id.py pypaimon/tests/table_update_by_row_id_chunked_test.py` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
