yangshangqing95 opened a new pull request, #17346:
URL: https://github.com/apache/iceberg/pull/17346
## Summary
Fixes #17345
Queries against the `all_manifests` metadata table can return incorrect
results when a filter contains a negated predicate on a column that cannot be
evaluated during snapshot planning.
For example, although delete manifests are present, the following queries
will incorrectly return no rows:
```sql
SELECT *
FROM catalog.db.table.all_manifests
WHERE content != 0;
```
```sql
SELECT *
FROM catalog.db.table.all_manifests
WHERE NOT(content = 0);
```
`AllManifestsTable.SnapshotEvaluator` is a may-match evaluator. Predicates
on columns other than `reference_snapshot_id` conservatively return `true`
because matching manifest rows may exist. Negating that result directly changes
`true` to `false` and incorrectly prunes the snapshot.
## Changes
- Rewrite `NOT` expressions before binding filters in
`AllManifestsTable.SnapshotEvaluator`.
- Preserve conservative evaluation for predicates that cannot be evaluated
during snapshot planning.
- Preserve snapshot pruning for predicates on `reference_snapshot_id`.
- Add core regression tests for:
- negated predicates on non-snapshot columns;
- negated `OR` predicates containing known and unknown values;
- negated `AND` predicates containing known and unknown values;
- negated snapshot-only predicates.
- Add Spark 3.5, 4.0, 4.1 SQL regression coverage for:
- `content != 0`;
- `content <> 0`;
- `NOT(content = 0)`.
## Why this fixes the issue
The evaluator previously handled an expression such as:
```text
NOT(content = 0)
```
as:
```text
content = 0 -> true because rows might match
NOT(content = 0) -> false
```
The `false` result caused the snapshot to be incorrectly pruned.
Rewriting the expression first converts it to:
```text
noteq
```
then leverage the noteq evaluator to do the filter.
The evaluator then conservatively keeps the snapshot because `content`
cannot be determined while planning snapshot tasks.
Rewriting also preserves valid pruning for predicates that only reference
`reference_snapshot_id`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
