LuciferYang opened a new pull request, #9527: URL: https://github.com/apache/paimon/pull/9527
### Purpose close #9526 Z-order encodes each order column into eight bytes and then interleaves the bits, and `ZOrderByteUtils.NULL_BYTES` is eight zero bytes standing for null. The boolean encoder writes only the first byte of its per-column buffer, and the remaining seven are never written, so they stay zero for the life of the indexer. FALSE therefore encoded to eight zero bytes, byte for byte identical to `NULL_BYTES`: a FALSE row and a NULL row got the same z-order key and were clustered as if the column held the same value, with nothing reporting a problem. FALSE now writes `0x01`, which keeps the unsigned order NULL, FALSE, TRUE and leaves TRUE at `0x81`. `SparkZOrderUDF.booleanToOrderedBytesUDF` carries its own copy of the encoding, so both are changed together. They have to agree: a column clustered by Spark and later compacted by Flink must land in the same order. ### Tests - `TestZOrderByteUtil.testBooleanDistinctFromNullSentinel` builds a two-boolean-column `ZIndexer` and asserts the three states are pairwise distinct and that the unsigned order really is NULL, then FALSE, then TRUE, using the `UnsignedBytes` comparator the file already uses. Interleaving two identical inputs is monotone in the input, so comparing the interleaved output compares the encodings. Pinning the order matters as much as the distinctness: `0x82` for FALSE would also be distinct while sorting FALSE above TRUE. - `SparkZOrderUDFTest.testBooleanColumnKeepsFalseOffTheNullSentinel` (new file) runs a local `SparkSession` over a nullable BOOLEAN column with true, false and null rows through `sortedLexicographically(col, BooleanType)` and asserts the three encodings are `8100000000000000`, `0100000000000000` and `0000000000000000`. It compares hex rather than the raw arrays on purpose: the UDF returns a per-column buffer it reuses for every row, so three collected `byte[]` values all alias one another and carry the last row's contents. The conversion happens inside the same projection, before the buffer is overwritten. Production is unaffected, because `ZorderSorter` feeds the result to `interleaveBits` within the same row's evaluation, which copies the bits into its own output buffer. Both fail against the pre-fix code: the indexer test on the FALSE-versus-NULL comparison, and the Spark test with FALSE coming back as `0000000000000000`. Note on running them: `TestZOrderByteUtil` is named with a `Test` prefix, and the root pom's `test.unit.pattern` is `**/*Test.*`, so that class runs in the integration-test execution rather than in `mvn test`. CI runs `mvn clean install` and reaches it; running it directly needs `-Dtest=TestZOrderByteUtil`, which gives 14 tests, 0 failures. `mvn -pl paimon-common test` on JDK 8: 12465 tests, 0 failures, 0 errors. `mvn -pl paimon-spark/paimon-spark-common -Dtest='SparkZOrderUDFTest,SortedIndexTopoBuilderTest' test`: 5 tests, 0 failures. checkstyle, spotless, enforcer and rat run clean on both modules. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
