LuciferYang opened a new issue, #9627: URL: https://github.com/apache/paimon/issues/9627
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark. Reached from `sys.compact` with `order_strategy => 'zorder'`, from write-path clustering, and from Spark's z-order sort, which calls the same two functions through `SparkZOrderUDF`. ### Minimal reproduce step Z-order clustering on a DOUBLE or FLOAT column that holds negative values. The transform that is supposed to make the bit pattern unsigned-comparable shifts by the wrong width: ```java public static ByteBuffer doubleToOrderedBytes(double val, ByteBuffer reuse) { ByteBuffer bytes = reuse(reuse, PRIMITIVE_BUFFER_SIZE); long lval = Double.doubleToLongBits(val); lval ^= ((lval >> (Integer.SIZE - 1)) | Long.MIN_VALUE); bytes.putLong(lval); return bytes; } ``` `lval` is 64 bits, so the mask has to come from an arithmetic shift of 63, not 31. Shifting a negative long right by 31 leaves the low 33 bits of the pattern in the mask instead of all ones, so only part of the magnitude is inverted: - `-2.5` gives `0x3ffbffff80080000` - `-2.5000000000000004`, a strictly smaller value, gives `0x3ffbffff80080001` Compared unsigned the smaller value sorts higher, and that holds for the majority of adjacent negative pairs inside each exponent block. `floatToOrderedBytes` widens to double and runs the same line, so floats are affected too. ### What doesn't meet your expectations? Values that sort one way must encode the other way round for the clustering to mean anything. With this transform, negative-valued FLOAT and DOUBLE columns get a z-value ordering that is scrambled within each exponent range, so clustering and the min/max statistics computed on the sorted output do not help skipping. Results stay correct, since a z-value is only a sort key. The same transform written correctly is a few files away, in `SortUtil`: `putFloatNormalizedKey` shifts an `int` by `Integer.SIZE - 1` and `putDoubleNormalizedKey` shifts a `long` by `Long.SIZE - 1`. The z-order copy kept the int-sized constant on a long. ### Anything else? The existing tests `testFloatOrdering` and `testDoubleOrdering` only feed `random.nextFloat()` and `random.nextDouble()`, which are in [0, 1), so no negative value has ever been exercised. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
