XiaoHongbo-Hope opened a new pull request, #9847: URL: https://github.com/apache/paimon/pull/9847
### Purpose Nested-column reconstruction currently converts offsets and null masks to Python lists, normalizes offsets element by element, and converts the result back to Arrow. This adds CPU and allocation overhead when reading large nested columns. Use Arrow subtract and if_else instead, preserving sliced offsets, null parents, the final offset, and int32/int64 offset types. File layout and commit behavior are unchanged. In an isolated two-file nested-column update benchmark, this change reduced median end-to-end elapsed time by about 18% across two unprofiled runs. This is workload-specific, not a general throughput guarantee. ### Tests - Regression coverage for List, LargeList and Map offsets, including nulls, empty arrays and slices. - Arrow 19.0.1: 160 tests passed across format_pyarrow_variant_row_group_test, table_update_by_row_id_chunked_test, variant_test and variant_path_test. - Project-configured Flake8 and git diff --check passed. - Older Arrow compatibility remains to be verified by CI. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
