cshuo opened a new pull request, #20517:
URL: https://github.com/apache/hudi/pull/20517

   ### Describe the issue this Pull Request addresses
   
   Closes #20508.
   
   `x IN (1, NULL)` can match `x = 1`, but the Flink statistics evaluator 
currently treats any NULL candidate as making the entire predicate impossible 
to match. If this expression reaches Hudi, matching files or partitions can be 
pruned.
   
   ### Summary and Changelog
   
   - Handle `IN` before the generic NULL-literal check and evaluate its 
non-null candidates. An all-NULL list still cannot match.
   - Return Java `null` for SQL NULL literals at the start of 
`getValueFromLiteral`, including literals with `DataTypes.NULL()`.
   - Add regression coverage for mixed and all-NULL lists, large lists, and 
both typed and untyped NULL literals.
   
   ### Impact
   
   Corrects statistics pruning for `IN` predicates containing NULL candidates. 
No storage-format, public API, or configuration changes.
   
   ### Risk Level
   
   Low: the evaluator behavior is covered directly by unit tests, and the 
existing expression conversion and predicate tests also pass.
   
   Validation with Flink 1.20: 67 tests passed across 
`TestExpressionEvaluators`, `TestExpressionUtils`, and 
`TestExpressionPredicates`.
   
   ```bash
   mvn test -Dflink1.20 
-Dtest=TestExpressionEvaluators,TestExpressionUtils,TestExpressionPredicates 
-Dcheckstyle.skip -Drat.skip=true -pl hudi-flink-datasource/hudi-flink
   ```
   
   Flink 1.20 rewrites the tested SQL `IN (..., NULL)` predicates before source 
pushdown, so SQL-level reproduction was not observed on that version. The 
regression tests construct the expressions directly to exercise the affected 
evaluator.
   
   ### Documentation Update
   
   None.
   
   ### Contributor's checklist
   
   - [ ] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to