Zouxxyy opened a new pull request, #1479:
URL: https://github.com/apache/paimon-rust/pull/1479

   ### Purpose
   
   Follow-up to #1189 and #1190 for file-level writer statistics. A Parquet 
file containing 5 and NaN can have footer bounds [5, 5]. Copying them to the 
manifest entry can cause another engine to skip the data file on an equality 
filter for NaN. Using partial comparisons for primary-key column statistics can 
also collapse the two signs of zero.
   
   ### Brief change log
   
   * Compute bounds for full/truncate FLOAT and DOUBLE columns from incoming 
Arrow batches, accumulating across writes and row groups. Other supported value 
columns continue to use the footer. Preserve counts/none modes and dense field 
mapping.
   * Share Java-order comparisons with the existing manifest partition 
predicate/collector and with the key column stats collector. Use typed Arrow 
arrays to select batch extrema.
   * Preserve physical sort order, merge identity and complete key endpoints. 
The committed-file test checks the scalar key/value bounds separately from the 
physical first and last composite keys.
   
   ### Tests
   
   Ran the column-statistics and manifest-partition-statistics unit tests, the 
Parquet unit suite, and the pk_value_stats_test and float_partition_filter_test 
integrations (119 tests total). Coverage includes positive and negative zero 
alone and together, NaN at either end, negative NaN, infinities, multiple 
batches and row groups, nulls and dense counts-only columns.
   
   ### API and Format
   
   The manifest schema and the Parquet file format are unchanged.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to