LuciferYang opened a new issue, #9514: URL: https://github.com/apache/paimon/issues/9514
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `9c7deebbd` (2.1-SNAPSHOT) ### Compute Engine Spark (`sys.compact` with `order_strategy => 'hilbert'`, and clustered writes) and Flink (`sort_compact`). Both engines have their own copy of the mapping: `HilbertIndexer` in paimon-common, `SparkHilbertUDF` in paimon-spark-common. ### Minimal reproduce step Hilbert clustering maps each order column to a long and then to a position on the curve. `PRIMITIVE_EMPTY`, which is `Long.MAX_VALUE`, is the value every type uses for null. The boolean mapping used it for TRUE: ```java // HilbertIndexer, BOOLEAN visitor return row.getBoolean(fieldIndex) ? PRIMITIVE_EMPTY : 0; // SparkHilbertUDF.booleanToOrderedLongUDF functions.udf((Boolean value) -> value ? PRIMITIVE_EMPTY : 0, DataTypes.LongType) ``` So a TRUE value and a NULL value produce the same curve position, and rows that differ on that column are clustered as if they were equal. The Spark side fails harder, because that UDF is also the only one of the nine in the file without a null check. A nullable boolean order column reaches the lambda as `null` and `value ?` unboxes it: ```sql CREATE TABLE T (a BOOLEAN, b INT) TBLPROPERTIES ('bucket' = '-1'); INSERT INTO T VALUES (true, 1), (false, 2), (null, 3); CALL paimon.sys.compact(table => 'T', order_strategy => 'hilbert', order_by => 'a,b'); ``` The compaction fails on the executor with a `NullPointerException` from the UDF. ### What doesn't meet your expectations? A boolean column has three states and each one should get its own position on the curve. TRUE sharing the null sentinel means the clustering quietly does less than it claims: files that could have been skipped on a `a = true` predicate are not, and nothing reports a problem. Clustering a nullable boolean column through Spark should also not fail at all. ### Anything else? This is the same class of defect as #7451 ("[spark] Fix NPE in SparkHilbertUDF and SparkZOrderUDF for null values"), which added the missing null guard to the string and binary UDFs in these two files and did not touch boolean. The hilbert value is a transient sort key, so nothing on disk records it. Fixing the mapping changes only how rows are laid out by a future sort-compact or clustered write. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
