JingsongLi opened a new pull request, #11303:
URL: https://github.com/apache/paimon/pull/11303

   ### Purpose
   
   The pure Python writer only populated manifest value statistics for 
`metadata.stats-mode=full`. As a result, an explicit `truncate(16)` on a table 
shared with Java produced empty value statistics on Python-written files.
   
   - Match Java's `truncate(16)` default and honor `none`, `counts`, `full`, 
column overrides, and primary-key level and changelog modes.
   - Reuse the Parquet writer's in-memory footer for scalar min/max bounds when 
available. Keep the Arrow fallback when a row group lacks reliable bounds, and 
preserve the existing handling of NaN and signed zero.
   - Apply the policy to ordinary data files, changelogs, and Data Evolution 
partial-column writes.
   
   Related to #10173.
   
   ### Tests
   
   - On current master, 61 focused Python tests passed using PyArrow 22.0.0, 
with 11 skipped. This covers manifest bounds, Parquet page indexes, changelog 
retry behavior, and REST writes.
   - The broader local regression run passed 1,568 tests. It excluded an 
existing pandas 3 dtype assertion reproduced on master and Ray cases requiring 
the optional dependency.
   - Local PyArrow 22 A/B benchmark on a 200,000-row, 64-column ZSTD append 
reduced statistics CPU time from 20.3 ms to 2.9 ms and median write-plus-commit 
time from 471.5 ms to 457.0 ms over six measured runs.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to