LuciferYang opened a new issue, #9526: URL: https://github.com/apache/paimon/issues/9526
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `9c7deebbd` (2.1-SNAPSHOT) ### Compute Engine Flink (`sort_compact` with `order_strategy = zorder`) and Spark (`sys.compact` with `order_strategy => 'zorder'`, and clustered writes). Each engine has its own copy of the encoding: `ZIndexer` in paimon-common, `SparkZOrderUDF` in paimon-spark-common. ### Minimal reproduce step Z-order encodes each order column into eight bytes and then interleaves the bits. `ZOrderByteUtils.NULL_BYTES` is eight zero bytes and stands for null. The boolean encoder writes only the first byte of its buffer: ```java // ZIndexer, BOOLEAN visitor ZOrderByteUtils.reuse(reuse, PRIMITIVE_BUFFER_SIZE); reuse.put(0, (byte) (row.getBoolean(fieldIndex) ? -127 : 0)); return reuse.array(); ``` The remaining seven bytes of that per-column buffer are never written, so they stay zero for the life of the indexer. FALSE therefore encodes to eight zero bytes, byte for byte identical to `NULL_BYTES`, and a FALSE row and a NULL row get the same z-order key: ```sql CREATE TABLE T (a BOOLEAN, b INT) TBLPROPERTIES ('bucket' = '-1'); INSERT INTO T VALUES (true, 1), (false, 2), (null, 3); CALL paimon.sys.compact(table => 'T', order_strategy => 'zorder', order_by => 'a,b'); ``` The compaction succeeds, and the rows with `a = false` and `a = null` are clustered as if that column held the same value. ### What doesn't meet your expectations? A boolean column has three states and each should get its own encoding, as it does for every other type in that class: TRUE is `0x81`, NULL is all zeros, and FALSE needs to be something else. Sharing the null encoding means the clustering silently does less than it claims: files that could be skipped for `a = false` are read anyway, and nothing reports a problem. ### Anything else? `SparkZOrderUDF.booleanToOrderedBytesUDF` has the same literal, so both engines are affected, and they have to stay in step: a column clustered by Spark and later compacted by Flink must land in the same order. While looking at this I also noticed that `NULL_BYTES` equals the encoding of `Long.MIN_VALUE` for a BIGINT or TIMESTAMP column, since `tinyintToOrderedBytes` and friends flip the sign bit. That one needs a different remedy and I am not proposing to change it here. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
