Zouxxyy opened a new pull request, #11317:
URL: https://github.com/apache/paimon/pull/11317

   ### Purpose
   
   Keep writer statistics sound for floating-point values across engines. 
Positive and negative zero are distinct under Java's ordering and all NaNs 
compare above positive infinity.
   
   * Compute Python partition and file bounds in Java order. Use the same 
ordering for sorting composite primary keys and grouping equal keys, while 
retaining the original column values. Determine the first and last key across 
the full file.
   * When Java copies an encoded manifest, project only its partition column to 
rebuild floating or incomplete statistics. Continue to copy the original Avro 
blocks. A complete-looking legacy range such as [5, 5] can hide a NaN 
partition, and an unknown null count must not become zero during copying.
   * When importing Parquet from disk, retain known floating null counts and 
omit floating bounds. External footers may omit NaNs. The normal Java writer 
continues to use its own in-memory metadata.
   
   Related to the writer side of #10492.
   
   ### Tests
   
   * Added regressions that copy an actual legacy Python manifest and import a 
PyArrow file containing 5, NaN and null. Exercised distinct signed zeros, 
positive zero alone, NaNs, nullable/composite partitions, key identity, and 
file/key/value statistics.
   * Ran seven Java suites (178 tests), seven Python suites (244 tests), 
Spotless, RAT and the Python file lint.
   * Recomputed bounds from the contents of 45 Python-written manifests. All 
stored partition bounds and null counts matched.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to