cshuo opened a new issue, #20449: URL: https://github.com/apache/hudi/issues/20449
## Description Several Spark data skipping issues can cause matching rows to be incorrectly pruned when predicates are evaluated against column or partition statistics. This issue tracks the related fixes and test coverage. The oracle-based tests in #20439 compare pruning results against the original predicates evaluated on raw rows. Any file or partition containing a matching row must remain a candidate. ## Related issues - [ ] Missing column statistics: #20296 (fix: #20328) - [ ] Prefix `LIKE` predicates: #20430 - [ ] Order-breaking casts and negative arithmetic factors: #19445 - [ ] Date/time transformations and format-sensitive parsing: #19481 - [ ] Partition stats pruning for time travel queries: #20443 - [ ] Oracle-based regression coverage for file and partition pruning: #20439 ## Gaps to track Create or link separate issues as needed for confirmed cases that are not fully covered above: - Arithmetic overflow and bitwise operations that do not preserve ordering. - Date/time transformations across DST transitions or with extreme values. - Transforms that produce NULL bounds from real values, such as logarithms on negative values. The related issues can be assigned and addressed independently, with the oracle tests helping validate the fixes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
