LuciferYang opened a new pull request, #9515:
URL: https://github.com/apache/paimon/pull/9515
### Purpose
close #9514
Hilbert clustering maps each order column to a long before placing it on the
curve, and `PRIMITIVE_EMPTY` (`Long.MAX_VALUE`) is the value every type uses
for null. The boolean mapping used it for TRUE, in both copies of the mapping:
`HilbertIndexer` in paimon-common and `SparkHilbertUDF.booleanToOrderedLongUDF`
in paimon-spark-common. A TRUE value and a NULL value therefore landed on the
same point of the curve, so rows that differ on that column clustered as if
they were equal, and a `a = true` predicate lost the file skipping the
clustering was supposed to buy. TRUE now maps to 1 and FALSE stays at 0 on both
sides.
The Spark UDF was also the only one of the nine in that file without a null
check. `functions.udf((Boolean value) -> ...)` has no input encoders, so
Spark's `HandleNullInputsForUDF` does not insert a null short circuit and the
lambda receives `null` directly, where `value ?` unboxed it into a
`NullPointerException` on the executor. Clustering a nullable boolean column
through `sys.compact` failed outright. #7451 added this guard to the string and
binary UDFs in the same two files and left boolean out.
The hilbert value is a transient sort key: `HilbertSorter` adds it, sorts on
it and drops it, on both engines. Nothing on disk records it, so existing
tables read back unchanged and only the layout produced by a future
sort-compact or clustered write differs.
### Tests
- `HilbertIndexerTest.testBooleanValuesDistinctFromNull` (new file) asserts
the curve position of a FALSE row, a TRUE row and a NULL row against
`hilbertCurvePosBytes` of `{0, 0}`, `{1, 1}` and `{MAX_VALUE, MAX_VALUE}`.
Pinning the exact positions pins the mapping, so an inverted mapping that keeps
the three distinct cannot pass and drift away from the Spark side; the three
pairwise-distinct assertions are kept as an explicit statement of the collision
itself.
-
`SparkHilbertUDFTest.testBooleanColumnMapsNullFalseAndTrueToDistinctValues`
(new file) runs a local `SparkSession` over a nullable BOOLEAN column with
true, false and null rows through `sortedLexicographically(col, BooleanType)`
and asserts null maps to `Long.MAX_VALUE`, TRUE to 1 and FALSE to 0. It lives
in paimon-spark-common next to `SortedIndexTopoBuilderTest`, which already
builds a local session there, and runs in about three seconds.
Verified red before the change: the indexer test fails on the exact-position
assertion, and the Spark test fails with `java.lang.NullPointerException`,
which also confirms that Spark really does hand `null` to that lambda.
`mvn -pl paimon-common test` on JDK 8: 12466 tests, 0 failures. `mvn -pl
paimon-spark/paimon-spark-common
-Dtest='SparkHilbertUDFTest,SortedIndexTopoBuilderTest' test`: 5 tests, 0
failures, both session-creating classes green in one JVM. checkstyle, spotless,
enforcer and rat run clean on both modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
