cshuo opened a new pull request, #20328: URL: https://github.com/apache/hudi/pull/20328
### Describe the issue this Pull Request addresses Closes https://github.com/apache/hudi/issues/20296. After adding columns to the column stats index, older files can have statistics for only some query columns. For example, a file containing `age = 25, name = 'alice'` may have age statistics but no name statistics. Spark can incorrectly discard that file for `age >= 0 AND name = 'alice'`: the translated name predicate evaluates to SQL NULL, and the whole-file fallback does not recover a file already represented in the index. ### Summary and Changelog - Split top-level AND predicates into independent conjuncts before translating them into column-statistics filters. Keep OR and NOT subtrees intact. - For each translated conjunct, identify the statistics it references and retain files when a corresponding null-count is NULL, indicating missing statistics. Apply this guard outside the entire conjunct, including negations. - Allow independent conjuncts with complete statistics to continue pruning. For example, `age > 100 AND name = 'alice'` can exclude a file with `age_max = 30` even when name statistics are missing. - Add candidate-file tests covering missing statistics, known all-null columns, complete statistics, unindexed files, AND/OR/NOT predicates, and combined versus separate query filters. - Add a Spark SQL regression that writes files before and after adding name to the indexed columns, verifies partial metadata coverage, and checks query results with data skipping enabled. ### Impact Corrects file pruning with partial column-statistics coverage while preserving pruning by independent predicates with complete statistics. No configuration, public API, or storage-format changes. The change is in the shared Spark candidate-file filtering path; the separate partition-statistics filtering path is unchanged. ### Risk Level Medium: the shared candidate-file filtering code is also used by column-statistics-based expression indexes, and this change affects boolean predicate handling. Local validation with Spark 3.5.9, Scala 2.12.18, and Java 11: - Targeted SQL regression in `TestDataSkippingQuery`: 1 passed. - `TestDataSkippingUtils`: 71 passed, including 20 parameterized missing-statistics cases. - `git diff --check`: passed. Validation used targeted source compilation with existing local module outputs and resolved dependencies. A full Maven reactor build, the full SQL suite, and dedicated expression-index integration tests were not run. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
