plusplusjiajia opened a new issue, #11164:
URL: https://github.com/apache/paimon/issues/11164

   ### Search before asking
   
   - [x] I searched in the [issues](https://github.com/apache/paimon/issues) 
and found nothing similar.
   
   ### Paimon version
   
   master (9b6c99719). PyPaimon at the same commit.
   
   ### Compute Engine
   
   Java clients (Spark, Flink) on JDK 8/11/17 and on JDK 19+, and PyPaimon.
   
   ### Minimal reproduce step
   
   Java names a FLOAT or DOUBLE partition with `value.toString()` 
(`InternalRowPartitionComputer` with legacy names, and 
`NumericToStringCastRule` otherwise). PyPaimon names it with 
`java_floating_string` in `pypaimon/utils/numeric_utils.py`, which follows the 
JDK 19+ `Float.toString` / `Double.toString` contract: the shortest decimal 
that round-trips.
   
   JDK 19 changed both methods 
([JDK-4511638](https://bugs.openjdk.org/browse/JDK-4511638)), so the same value 
gets different digits:
   
   | Value | Java 11 `toString` | PyPaimon (JDK 19+ form) |
   |---|---|---|
   | FLOAT bits `0x527a6863` | `2.68873286E11` | `2.6887329E11` |
   | FLOAT bits `0x4f083ce4` | `2.2856919E9` | `2.285692E9` |
   | DOUBLE `1.0E23` | `9.999999999999999E22` | `1.0E23` |
   
   Repro on Java 11:
   
   ```java
   System.out.println(Float.toString(Float.intBitsToFloat(0x527a6863))); // 
2.68873286E11
   ```
   
   ```python
   import struct
   from pypaimon.utils.numeric_utils import java_floating_string
   print(java_floating_string(struct.unpack('>f', 
bytes.fromhex('527a6863'))[0], True))  # 2.6887329E11
   ```
   
   Across 200,000 random bit patterns, Java 11 and PyPaimon disagree on 10.9% 
of FLOAT values and 0.3% of DOUBLE values.
   
   ### What doesn't meet your expectations?
   
   A FLOAT or DOUBLE partition written by one client can land in a different 
directory than the same value written by another. A Java equality filter or 
static overwrite builds the exact directory name, so it misses PyPaimon's 
partition, and the reverse. This applies to normal tables and format tables 
alike, since both use the same formatting.
   
   Java is not consistent with itself either: Paimon writers on JDK 8/11/17 and 
on JDK 19+ already name these partitions differently, so PyPaimon cannot match 
every Java writer.
   
   ### Anything else?
   
   Possible directions:
   
   1. **PyPaimon matches JDK 8/11/17.** Port the pre-JDK-19 `FloatingDecimal` 
digits for FLOAT and DOUBLE partition names. `numeric_utils.py` already has a 
pre-JDK-19 path for doubles (`java_double_decimal_text`). This matches what 
most Spark and Flink deployments run today, but not JDK 19+ writers.
   2. **Keep the JDK 19+ form** and document the difference.
   3. **Make Java JDK-independent.** Format FLOAT and DOUBLE partition values 
with a fixed algorithm instead of `toString()`, so that every JDK and PyPaimon 
agree. This is the cleanest, but it changes directory names that some existing 
Java writers produce.
   
   Options 1 and 3 change directory names that existing writers already 
produce, so partitions written before the change would no longer match filters 
and would need to be rewritten. FLOAT and DOUBLE partition keys are uncommon, 
which limits the impact.
   
   Found while reviewing #11018.
   
   ### Are you willing to submit a PR?
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to