XiaoHongbo-Hope opened a new pull request, #11175: URL: https://github.com/apache/paimon/pull/11175
### Purpose Speed up pure Python shredded VARIANT writes by reusing parsed metadata within each Arrow batch, decoding object ID/offset tables in bulk, and preserving the existing key order when rebuilding residual values. The on-disk format and public API are unchanged. For a local synthetic write of 64 rows with 5,362 fields and 200 shredded fields, median write-and-commit time fell from 1.194 s to 0.461 s (PyArrow 22). The generated shredded Parquet bytes have the same SHA-256 before and after. This is not a customer-table benchmark. ### Tests - Variant and PyArrow reader tests: 244 passed, 5 skipped, 97 subtests passed. - Flake8 and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
