LuciferYang opened a new issue, #10553: URL: https://github.com/apache/paimon/issues/10553
### Search before asking - [X] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master (726675eac7) ### Compute Engine Spark (DATE, FLOAT, DOUBLE) and Flink (FLOAT, DOUBLE) ### Minimal reproduce step DATE column on Spark: ```sql CREATE TABLE T (id INT, dt DATE) TBLPROPERTIES ('bucket' = '-1'); INSERT INTO T VALUES (1, DATE '2024-01-03'), (2, DATE '1969-12-31'); CALL sys.compact(table => 'T', order_strategy => 'hilbert', order_by => 'dt,id'); ``` DOUBLE columns, either engine: hilbert-sort a table whose `x, y` columns hold both negative and positive values. ### What doesn't meet your expectations? DATE on Spark: `SparkHilbertUDF` casts the date to bigint. With ANSI mode on, the default in Spark 4, the compact fails at analysis with `DATATYPE_MISMATCH.CAST_WITH_FUNC_SUGGESTION: cannot cast "DATE" to "BIGINT"`. With ANSI off the cast yields null for every row, so the date column does not take part in the clustering. FLOAT and DOUBLE: both `HilbertIndexer` and `SparkHilbertUDF` pass `Double.doubleToLongBits` to a curve built with `HilbertCurve.bits(63)`, which reads only the low 63 bits of each coordinate. The sign bit is dropped, so `x` and `-x` get the same curve position, and points from both signs of the plane end up interleaved in the sorted output. ### Anything else? _No response_ ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
