LuciferYang opened a new issue, #9625: URL: https://github.com/apache/paimon/issues/9625
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Flink and the Java API, wherever clustering or sort-compact runs: `append/cluster/ZorderSorter` on the write path and `CALL sys.compact(..., order_strategy => 'zorder', ...)`. ### Minimal reproduce step Cluster a table by a DECIMAL column and compare the z-values two rows get. `ZIndexer`'s decimal branch wrote the unscaled value as a minimal two's-complement byte array into the fixed 8-byte z-value buffer: ```java return o == null ? NULL_BYTES : ZOrderByteUtils.byteTruncateOrFill( ((Decimal) o).toUnscaledBytes(), PRIMITIVE_BUFFER_SIZE, reuse) .array(); ``` `byteTruncateOrFill` left-aligns and pads with 0x00, and z-values are compared unsigned, so for DECIMAL(20,2): - `-1.00` gives unscaled -100, one byte `0x9C`, padded to `9C 00 00 00 00 00 00 00` - `1.00` gives unscaled 100, one byte `0x64`, padded to `64 00 00 00 00 00 00 00` `0x9C` is 156 unsigned and `0x64` is 100, so -1.00 sorts above 1.00. Length makes it worse: `100.00` is unscaled 10000, two bytes `27 10`, which sorts below the one-byte `1.00`. And `0.00` gives a single `0x00` byte padded to eight zero bytes, which is byte-for-byte the `NULL_BYTES` sentinel, so zero and NULL are indistinguishable. `toUnscaledBytes` is `toBigDecimal().unscaledValue().toByteArray()` for every precision, so this is not limited to the wide decimals. ### What doesn't meet your expectations? Clustering by a DECIMAL column should put nearby values near each other. Instead the ordering is scrambled inside each byte-length class and reversed across the sign, so the clustering does not help data skipping, and min/max statistics on the sorted output are no better than random. Results stay correct, since a z-value is only a sort key, but the feature does nothing useful for decimals. Every other numeric type in that visitor goes through the sign-flipped fixed-width transform in `ZOrderByteUtils` (`intToOrderedBytes`, `longToOrderedBytes`, and so on), which is order-preserving under unsigned comparison and keeps zero away from the null sentinel. The decimal branch is the one that does not. ### Anything else? `HilbertIndexer` encodes decimals as `toBigDecimal().longValue()`, which drops the fractional part entirely, so every value with magnitude below 1 collapses to the same key. Different bug, same area. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
