cshuo opened a new pull request, #20512:
URL: https://github.com/apache/hudi/pull/20512
### Describe the issue this Pull Request addresses
Closes #20507.
Flink data skipping evaluates a `NOT` expression by negating its child
evaluator's result. The child result means that a file or partition **may**
contain matching records; it does not mean that all records match.
For example, a file with `age` values `{23, 33}` has statistics `[23, 33]`.
The evaluator for `age = 23` returns `true`, so the current `NOT` evaluator
returns `false` and excludes the file, even though `age = 33` satisfies
`NOT(age = 23)`.
### Summary and Changelog
- Conservatively retain candidates for `NOT` expressions instead of negating
the child evaluator's result.
- Add evaluator tests for mixed values, nulls, nested `NOT`, and
combinations with `AND` and `OR`.
- Add an integration test that writes a file with `age` statistics `[23,
33]` and verifies that an explicitly constructed `NOT(age = 23)` expression
does not exclude the file.
### Flink SQL optimizer behavior
Before pushing filters to Hudi, Flink normalizes common `NOT` predicates
into equivalent expressions. These rules cover the comparison operators, null
checks, `IN`, and boolean combinations recognized by
`ExpressionUtils.isSimpleCallExpression`:
| SQL predicate | Normalized expression |
| --- | --- |
| `NOT(age = 23)` | `age <> 23` |
| `NOT(age <> 23)` | `age = 23` |
| `NOT(age < 23)` | `age >= 23` |
| `NOT(age <= 23)` | `age > 23` |
| `NOT(age > 23)` | `age <= 23` |
| `NOT(age >= 23)` | `age < 23` |
| `NOT(age IS NULL)` | `age IS NOT NULL` |
| `NOT(age IS NOT NULL)` | `age IS NULL` |
| `NOT(age IN (23, 25, 27))` | `age <> 23 AND age <> 25 AND age <> 27` |
| `NOT(NOT(age = 23))` | `age = 23` |
| `NOT(A AND B)` | `NOT A OR NOT B` |
| `NOT(A OR B)` | `NOT A AND NOT B` |
As a result, common SQL predicates can still benefit from statistics-based
pruning through the existing evaluators. This change provides a conservative
fallback if an unrewritten `NOT` expression reaches Hudi.
### Impact
Unrewritten `NOT` expressions no longer cause data skipping to incorrectly
exclude candidates. Such expressions no longer contribute to statistics-based
pruning, while other conjunctive filters can still prune candidates.
### Risk Level
Low. The change favors retaining candidates when a `NOT` expression reaches
the statistics evaluator. This can reduce pruning efficiency for those
expressions.
### Documentation Update
None.
### Test Plan
- `TestExpressionEvaluators` — 20 tests passed.
- `ITTestHoodieDataSource#testDataSkippingWithNotExpression` — passed with
Flink 2.2.1.
### Contributor's checklist
- [ ] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
