mbutrovich opened a new issue, #19040:
URL: https://github.com/apache/iceberg/issues/19040

   ### Apache Iceberg version
   
   main (development)
   
   ### Query engine
   
   None
   
   ### Please describe the bug 🐞
   
   The [sorting 
rule](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/format/spec.md?plain=1#L730)
 orders floats as `-NaN` < `-Infinity` < `-value` < `-0` < `0` < `value` < 
`Infinity` < `NaN` and says this "aligns with the implementation of Java 
floating-point types comparisons." Java's 
[`Double.compareTo`](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/Double.html#compareTo(java.lang.Double))
 treats every NaN as equal and greater than `+Infinity`, so Java has no `-NaN` 
at the low end. The listed order is IEEE 754 `totalOrder`.
   
   Implementations follow different halves of the sentence. Iceberg Java 
compares floats with 
[`Comparator.naturalOrder()`](https://github.com/apache/iceberg/blob/2bd60a093a889a44a9e67033dba3abf4c36895f5/api/src/main/java/org/apache/iceberg/types/Comparators.java#L40-L41),
 which is `Double.compareTo`. iceberg-rust uses 
[`total_cmp`](https://github.com/apache/iceberg-rust/blob/641f59049e0aefad0c60e19915ecdb54933e002f/crates/iceberg/src/spec/values/datum.rs#L168-L176),
 which puts negative NaNs first and orders NaNs by payload.
   
   Following `totalOrder` also makes a computed NaN's position depend on 
hardware. IEEE 754 doesn't specify the sign of a NaN result. For `0.0 / 0.0`, 
x86 SSE returns `0xfff8000000000000` and AArch64 returns `0x7ff8000000000000`, 
so the same expression sorts first on one writer and last on another. This 
matters most where position must be exact, such as the ordering keys in the 
secondary index spec (#16961), which cites this rule.
   
   #2891 proposed removing `-NaN` and grouping all NaNs after `Infinity`. It 
was closed without a decision. The concern there was that requiring writers to 
canonicalize NaN is extra work and discards legitimate NaN bit patterns.
   
   Could the rule say that all NaN representations compare equal and sort after 
`Infinity`? This matches Java and the sentence that already claims it does. It 
constrains only comparison, so stored values keep their bits and writers don't 
canonicalize.
   
   ### Willingness to contribute
   
   - [ ] I can contribute a fix for this bug independently
   - [x] I would be willing to contribute a fix for this bug with guidance from 
the Iceberg community
   - [ ] I cannot contribute a fix for this bug at this time


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to