GGraziadei opened a new pull request, #17669:
URL: https://github.com/apache/iceberg/pull/17669
Closes #17658.
Two independent defects in the Spark Z-order clustering path, both spotted
by @manuzhang while
reviewing #16827. They are fixed here rather than in that PR because they
change existing Z-order
behaviour and deserve their own review. Both are present in `spark/v3.5`,
`spark/v4.0` and
`spark/v4.1`, so all three are fixed together.
### 1. NPE when a boolean clustering column contains nulls
`SparkZOrderUDF.booleanToOrderedBytesUDF()` unboxed a possibly-null
`Boolean` with no guard, so a
single null aborted the whole rewrite with:
```
[FAILED_EXECUTE_UDF] ... NullPointerException:
Cannot invoke "java.lang.Boolean.booleanValue()" because "value" is null
```
Boolean is the **only** converter with this problem — the seven numeric and
temporal converters
already return `PRIMITIVE_EMPTY` for null, and `stringToOrderedBytes` /
`byteTruncateOrFill` handle
it inside `ZOrderByteUtils`. The surrounding code therefore already
establishes the semantics — a
null sorts lowest, as all-zero bytes — and this change simply makes boolean
follow it. No type that
works today changes behaviour.
### 2. Case-insensitive validation kept the caller's spelling
`validZOrderColNames` resolved the column case-insensitively when
`spark.sql.caseSensitive` is
`false` (the Spark default) but then stored the caller's spelling instead of
the resolved field
name. Validation passed, and the mis-cased name later reached the
case-sensitive
`df.schema().apply(...)` in `zValue`, failing with:
```
[FIELD_NOT_FOUND] No such struct field `C2` in `c1`, `c2`, `c3`. SQLSTATE:
42704
```
Z-ordering on `C2` in a table whose column is `c2` now works, which is what
case-insensitive
resolution promised.
### Tests
Three regression tests were added to `TestRewriteDataFilesAction` in each
Spark version. Each was
written first and confirmed to fail for the expected reason before the fix:
- `zOrderUDFEncodesNullValuesAsZeroBytes` — asserts that a null of every one
of the 12 supported
types produces all-zero ordered bytes, pinning the null-sorts-lowest
convention across types
rather than only checking that boolean no longer throws;
- `zOrderSortWithNullBooleanValues` — end-to-end rewrite over a table with
null booleans;
- `zOrderSortWithMismatchedColumnCase` — end-to-end rewrite using `C2`/`C3`
against columns
`c2`/`c3`.
Full `TestRewriteDataFilesAction` suite passes on all three versions (549
tests total, 0 failures),
along with `checkstyleMain`, `checkstyleTest` and `spotlessCheck`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
