cshuo opened a new pull request, #20517: URL: https://github.com/apache/hudi/pull/20517
### Describe the issue this Pull Request addresses Closes #20508. `x IN (1, NULL)` can match `x = 1`, but the Flink statistics evaluator currently treats any NULL candidate as making the entire predicate impossible to match. If this expression reaches Hudi, matching files or partitions can be pruned. ### Summary and Changelog - Handle `IN` before the generic NULL-literal check and evaluate its non-null candidates. An all-NULL list still cannot match. - Return Java `null` for SQL NULL literals at the start of `getValueFromLiteral`, including literals with `DataTypes.NULL()`. - Add regression coverage for mixed and all-NULL lists, large lists, and both typed and untyped NULL literals. ### Impact Corrects statistics pruning for `IN` predicates containing NULL candidates. No storage-format, public API, or configuration changes. ### Risk Level Low: the evaluator behavior is covered directly by unit tests, and the existing expression conversion and predicate tests also pass. Validation with Flink 1.20: 67 tests passed across `TestExpressionEvaluators`, `TestExpressionUtils`, and `TestExpressionPredicates`. ```bash mvn test -Dflink1.20 -Dtest=TestExpressionEvaluators,TestExpressionUtils,TestExpressionPredicates -Dcheckstyle.skip -Drat.skip=true -pl hudi-flink-datasource/hudi-flink ``` Flink 1.20 rewrites the tested SQL `IN (..., NULL)` predicates before source pushdown, so SQL-level reproduction was not observed on that version. The regression tests construct the expressions directly to exercise the affected evaluator. ### Documentation Update None. ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
