SEPURI-SAI-KRISHNA opened a new pull request, #19646:
URL: https://github.com/apache/hudi/pull/19646
### Describe the issue this Pull Request addresses
Data skipping prunes files it must keep when a query filters on `IS NULL`
(or `<=> null`)
and the column-stats index holds no stats for that column in that file.
Closes #19645.
`DataSkippingUtils` translated:
* `colA is null` → `colA_nullCount > 0`
* `colA <=> null` → `colA_nullCount = null`
The transposed column-stats index fills `min`/`max`/`nullCount` with `null`
when a file has
no stats record for a column — a column added by schema evolution (older
files hold no
record for it) or a column whose type the index does not support. For such
rows the first
predicate evaluates to `null` and the second is `null` for every row, so
`indexDf.where(indexFilter)` drops them and the corresponding files are
pruned. Records in
those files are all null for that column, i.e. exactly the rows the query
asks for, so the
query silently returns fewer rows than it should.
The `is not null` translation right below already handles the unknown-stats
case
(`colA_nullCount = null or colA_valueCount = null or ...`), and the fill-in
code in
`ColumnStatsIndexSupport#transpose` assumes "the filter includes an isNull
check" — which
held only for `is not null`.
### Summary and Changelog
* `DataSkippingUtils`: `IsNull` and `EqualNullSafe(col, null)` now translate
to
`colA_nullCount is null OR colA_nullCount > 0`, via a new
`ColumnStatsExpressionUtils#genColumnIsNullExpression` helper. A file
whose null-count is
unknown is kept instead of being pruned; when the null-count is known the
pruning
decision is unchanged.
* `EqualNullSafe(col, null)` previously produced `colA_nullCount = null`, an
always-null
predicate that prunes every indexed file; it now shares the `IS NULL`
translation, which
is what `<=> null` means. (Spark's `NullPropagation` usually rewrites `col
<=> null` into
`col is null` before the filter reaches `HoodieFileIndex`, so this arm is
mostly latent —
but it is reached by callers that translate un-optimized expressions.)
* Comments on both arms (and the mirrored ones in
`containsNullOrValueCountBasedFilters`)
now describe the actual translation.
* `TestDataSkippingUtils`: three cases added to
`testSupportedAndUnsupportedDataSkippingColumnsSource` (the suite that
covers columns
without stats) — `B is null`, `B <=> null` and `A = 1 and B is null` over
an index where
one file has no stats for `B`. All three fail on master and pass with this
change.
No code was copied.
### Impact
Fixes silently incomplete results for `IS NULL` / `<=> null` queries on
Spark when data
skipping is enabled and the queried column has no stats in some files (most
commonly after
`ADD COLUMN` schema evolution). Files with unknown null-counts are now
scanned, so such
queries read a few more files than before — a correctness-required
trade-off; queries on
columns that are fully indexed are unaffected.
No public API, config or storage-format change.
### Risk Level
low
The change only relaxes a pruning predicate (strictly fewer files are
pruned), so it cannot
introduce new false pruning. Verified with the new and existing
`TestDataSkippingUtils`
cases.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
