plusplusjiajia opened a new pull request, #10872: URL: https://github.com/apache/paimon/pull/10872
### Purpose Decimals with precision above 18 store their unscaled value as two's complement bytes. PyPaimon computed the length as `(bit_length() + 8) // 8`, which adds an extra byte for negative values that are exactly -2^(8k-1). For example, -1.28 in `DECIMAL(20, 2)` was written as `ff80`, while Java writes `80` (`BigInteger.toByteArray`). Such primary keys hashed to a different bucket than in Java (bucket 2 instead of 7 with 10 buckets), and such partition values had different bytes than Java's. `BinaryRow` serialization and the row format writer now use the minimal encoding, like Java, through one shared helper. The global index key serializer also uses this helper instead of its own copy. ### Tests - `DecimalTest#test_decimal_not_compact_java_bytes` and `#test_decimal_fixed_bucket_matches_java` in `decimal_test.py`: `BinaryRow` bytes and the fixed bucket match Java, and longer encodings are still read. - `TestFormatRowReaderWriter#test_decimal_minimal_bytes`: the row format writer writes Java's bytes. - `DecimalKeySerializerTest#test_java_decimal_bytes` and `test_decimal.py`: the global index key serializer and `Decimal` keep Java's bytes. - 7 of the new test cases fail on master; the key serializer and `Decimal` cases guard existing behavior. The related tests pass locally: 104 tests. flake8 is clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
