SEPURI-SAI-KRISHNA opened a new issue, #19645:
URL: https://github.com/apache/hudi/issues/19645
## Bug Description
**What happened:**
`DataSkippingUtils` translates `colA IS NULL` into the column-stats-index
predicate
`colA_nullCount > 0`, and `colA <=> null` into `colA_nullCount = null`.
The transposed column-stats index is not guaranteed to hold stats for every
column of
every file. `ColumnStatsIndexSupport#transpose` explicitly fills
`min`/`max`/`nullCount`
with `null` when a file has no stats record for a column, which happens when:
1. the column was added by schema evolution (files written before the `ADD
COLUMN` hold no
stats record for it — the behavior codified in #18807), or
2. the column's type is not supported by the index.
For those rows:
* `colA_nullCount > 0` evaluates to `null` → `indexDf.where(...)` in
`SparkBaseIndexSupport` drops the row → the file is pruned, even though
every one of its
records has `colA = null`. Rows are silently missing from the query result.
* `colA_nullCount = null` is `null` for *every* row (SQL `= null` is never
true), so
`colA <=> null` prunes every indexed file. (Spark's `NullPropagation`
normally rewrites
`col <=> null` into `col is null` before the filter reaches
`HoodieFileIndex`, so this
arm is mostly latent, but it is wrong wherever it is reached.)
The neighbouring `IS NOT NULL` translation already guards against exactly
this
(`colA_nullCount = null or colA_valueCount = null or colA_nullCount <
colA_valueCount`),
and `ColumnStatsIndexSupport#transpose` justifies its `null` fill-in with
"because the
filter includes an isNull check" — which holds only for `IS NOT NULL`, not
for `IS NULL`.
So this looks like a gap rather than a deliberate trade-off.
**What you expected:**
A file whose null-count for the queried column is unknown must not be
pruned. `IS NULL` and
`<=> null` should return the same rows with and without data skipping.
**Steps to reproduce:**
Verified at the translation level, by adding cases to
`TestDataSkippingUtils.testSupportedAndUnsupportedDataSkippingColumnsSource`
— the suite
that already covers columns without stats — over an index where `file_1` has
no stats for
`B` (`B_minValue`/`B_maxValue`/`B_nullCount` all `null`), `file_2` has
`B_nullCount = 0`
and `file_3` has `B_nullCount = 1`:
| filter | files that must survive pruning | files current master returns |
|---|---|---|
| `B is null` | `file_1, file_3` | `file_3` |
| `B <=> null` | `file_1, file_3` | *(none)* |
| `A = 1 and B is null` | `file_1` | *(none)* |
The user-facing shape of this, which follows from the above but which I have
not run
end-to-end: on a table with metadata + column stats enabled (the default
indexes all
columns), write a batch, `ALTER TABLE t ADD COLUMNS (c string)`, write a
second batch, then
`SELECT * FROM t WHERE c IS NULL` with `hoodie.enable.data.skipping=true`.
The records
written before the `ADD COLUMN` — all of which have `c = null` — should be
absent from the
result and present when data skipping is disabled.
**Suggested fix:**
Translate both arms to `colA_nullCount IS NULL OR colA_nullCount > 0`, so a
file whose
null-count is unknown is kept while pruning stays unchanged when the
null-count is known.
I have this working with the tests above and will open a PR referencing this
issue.
## Environment
**Hudi version:** master (1.3.0-SNAPSHOT); the translation predates it by
several releases
**Query engine:** Spark (3.5)
**Relevant configs:** `hoodie.metadata.enable=true`,
`hoodie.metadata.index.column.stats.enable=true`,
`hoodie.enable.data.skipping=true`
## Logs and Stack Trace
None — nothing fails, the results are just silently incomplete.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
