plusplusjiajia opened a new issue, #11164: URL: https://github.com/apache/paimon/issues/11164
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (9b6c99719). PyPaimon at the same commit. ### Compute Engine Java clients (Spark, Flink) on JDK 8/11/17 and on JDK 19+, and PyPaimon. ### Minimal reproduce step Java names a FLOAT or DOUBLE partition with `value.toString()` (`InternalRowPartitionComputer` with legacy names, and `NumericToStringCastRule` otherwise). PyPaimon names it with `java_floating_string` in `pypaimon/utils/numeric_utils.py`, which follows the JDK 19+ `Float.toString` / `Double.toString` contract: the shortest decimal that round-trips. JDK 19 changed both methods ([JDK-4511638](https://bugs.openjdk.org/browse/JDK-4511638)), so the same value gets different digits: | Value | Java 11 `toString` | PyPaimon (JDK 19+ form) | |---|---|---| | FLOAT bits `0x527a6863` | `2.68873286E11` | `2.6887329E11` | | FLOAT bits `0x4f083ce4` | `2.2856919E9` | `2.285692E9` | | DOUBLE `1.0E23` | `9.999999999999999E22` | `1.0E23` | Repro on Java 11: ```java System.out.println(Float.toString(Float.intBitsToFloat(0x527a6863))); // 2.68873286E11 ``` ```python import struct from pypaimon.utils.numeric_utils import java_floating_string print(java_floating_string(struct.unpack('>f', bytes.fromhex('527a6863'))[0], True)) # 2.6887329E11 ``` Across 200,000 random bit patterns, Java 11 and PyPaimon disagree on 10.9% of FLOAT values and 0.3% of DOUBLE values. ### What doesn't meet your expectations? A FLOAT or DOUBLE partition written by one client can land in a different directory than the same value written by another. A Java equality filter or static overwrite builds the exact directory name, so it misses PyPaimon's partition, and the reverse. This applies to normal tables and format tables alike, since both use the same formatting. Java is not consistent with itself either: Paimon writers on JDK 8/11/17 and on JDK 19+ already name these partitions differently, so PyPaimon cannot match every Java writer. ### Anything else? Possible directions: 1. **PyPaimon matches JDK 8/11/17.** Port the pre-JDK-19 `FloatingDecimal` digits for FLOAT and DOUBLE partition names. `numeric_utils.py` already has a pre-JDK-19 path for doubles (`java_double_decimal_text`). This matches what most Spark and Flink deployments run today, but not JDK 19+ writers. 2. **Keep the JDK 19+ form** and document the difference. 3. **Make Java JDK-independent.** Format FLOAT and DOUBLE partition values with a fixed algorithm instead of `toString()`, so that every JDK and PyPaimon agree. This is the cleanest, but it changes directory names that some existing Java writers produce. Options 1 and 3 change directory names that existing writers already produce, so partitions written before the change would no longer match filters and would need to be rewritten. FLOAT and DOUBLE partition keys are uncommon, which limits the impact. Found while reviewing #11018. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
