SEPURI-SAI-KRISHNA opened a new issue, #17655:
URL: https://github.com/apache/iceberg/issues/17655
### Apache Iceberg version
main (development)
### Query engine
Other
### Please describe the bug 🐞
`ParquetDictionaryRowGroupFilter.notStartsWith` can skip row groups that
contain
matching rows, silently dropping those rows from scan results.
Verified on `main` at `cd758fcda`. Not a regression — `notStartsWith` is
unchanged
in `1.11.0`, so released versions are affected the same way.
The filter decides whether a row group can be skipped by inspecting only the
column's dictionary. A Parquet dictionary contains non-null values only, so
when
every dictionary entry starts with the prefix the filter returns
`ROWS_CANNOT_MATCH` — even when the column also contains nulls.
In Iceberg a null value matches `notStartsWith`: `Evaluator` implements it as
`!startsWith(...)`, and `startsWith` evaluates to false for null. A row group
holding nulls therefore does contain matching rows and must not be skipped.
### Reproduce
Write a Parquet file with a dictionary-encoded optional string column where
every
non-null value shares a prefix and some rows are null, then scan with a
`notStartsWith` filter on that prefix:
```java
Expression filter = Expressions.notStartsWith("some_nulls", "some");
```
The rows with a null `some_nulls` are missing from the result. Reading the
same
data without dictionary encoding (or through the metrics filter alone)
returns
them.
### Expected
The row group is read, and the null rows are returned.
### Why this is inconsistent with the rest of the codebase
Every other evaluator already handles this:
- `InclusiveEvalVisitor#notStartsWith` returns `ROWS_MIGHT_MATCH` when
`mayContainNull(id)`
- `ParquetMetricsRowGroupFilter#notStartsWith` returns `ROWS_MIGHT_MATCH`
when
`mayContainNull(colStats)`
- `ParquetBloomRowGroupFilter` and ORC's `ExpressionToSearchArgument` do not
push
the predicate down at all
And within `ParquetDictionaryRowGroupFilter` itself, `notEq`, `notIn` and
`notNaN` all consult `mayContainNulls` for exactly this reason (#6431,
#6836).
`notStartsWith` is the only negated predicate missing the check.
The existing unit test asserts the current behavior, so this is not caught
today:
```java
new ParquetDictionaryRowGroupFilter(SCHEMA, notStartsWith("some_nulls",
"some"))
.shouldRead(parquetSchema, rowGroupMetadata, dictionaryStore);
// asserted isFalse()
```
`some_nulls` is written as `(i % 10 == 0) ? null : "some"`, so that row
group does
contain matching (null) rows. `data`'s `TestMetricsRowGroupFilter` asserts
the
opposite for the metrics filter on a comparable column.
### Note on engine-visible impact
In Spark SQL, `NOT (col LIKE 'x%')` yields NULL for a null `col` and Spark
re-applies the filter after the scan, so Spark SQL users typically will not
observe missing rows. The data loss is directly observable through Iceberg's
own
API, e.g.
`IcebergGenerics.read(table).where(Expressions.notStartsWith(...))`,
and in any engine that trusts the pushdown. Independently of that, the same
query
returning different results depending on whether a column happened to be
dictionary-encoded is the core defect.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
