XiaoHongbo-Hope opened a new pull request, #11175:
URL: https://github.com/apache/paimon/pull/11175

   ### Purpose
   
   Speed up pure Python shredded VARIANT writes by reusing parsed metadata 
within each Arrow batch, decoding object ID/offset tables in bulk, and 
preserving the existing key order when rebuilding residual values. The on-disk 
format and public API are unchanged.
   
   For a local synthetic write of 64 rows with 5,362 fields and 200 shredded 
fields, median write-and-commit time fell from 1.194 s to 0.461 s (PyArrow 22). 
The generated shredded Parquet bytes have the same SHA-256 before and after. 
This is not a customer-table benchmark.
   
   ### Tests
   
   - Variant and PyArrow reader tests: 244 passed, 5 skipped, 97 subtests 
passed.
   - Flake8 and `git diff --check` passed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to