cshuo opened a new issue, #20507: URL: https://github.com/apache/hudi/issues/20507
## Description Flink data skipping evaluates `NOT` by negating its child evaluator's result. However, a child evaluator returns whether a file **may contain** matching records, not whether all records in the file match. Negating that result can incorrectly exclude files that contain records satisfying the `NOT` predicate. This can affect file and partition pruning when a `NOT` expression reaches the statistics evaluator. ## Example Consider a file containing two records: | Record | `age` | |--------|------:| | `id1` | 23 | | `id2` | 33 | The file's column statistics are `min(age) = 23` and `max(age) = 33`. For the predicate: ```sql NOT (age = 23) ``` the current evaluator proceeds as follows: 1. `age = 23` returns `true` because 23 falls within `[23, 33]`. 2. `NOT` negates that result and returns `false`. 3. Data skipping excludes the file, even though the record with `age = 33` satisfies the predicate. ## Reproduction and scope A local integration test writes the records above to a Hudi table and verifies the actual file statistics. It then constructs a `NOT(age = 23)` expression and passes it to the file statistics index. The index returns an empty candidate set, although the file must be retained: ```text expected: [<file>.parquet] actual: [] ``` The SQL query `WHERE NOT (age = 23)` itself returned the correct result in the tested Flink 2.2 setup: Flink rewrote the predicate to `age <> 23` before it reached Hudi. The reproduction therefore confirms the statistics evaluator defect, but does not demonstrate an end-to-end SQL failure for that query. Further investigation is needed to identify SQL expressions and Flink versions that pass a `NOT` expression through to Hudi. ## Expected behavior A file or partition should be pruned only when its statistics prove that no record can satisfy the predicate. When a `NOT` expression cannot be evaluated safely, data skipping should conservatively retain the candidate. ## Relevant code - `ExpressionEvaluators.Not.eval` directly negates the child evaluator result: https://github.com/apache/hudi/blob/b0d16f94b3b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/ExpressionEvaluators.java#L462-L474 - `ExpressionUtils.isSimpleCallExpression` accepts `NOT` expressions: https://github.com/apache/hudi/blob/b0d16f94b3b/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/util/ExpressionUtils.java#L72-L87 Related to umbrella issue #20500. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
