voonhous opened a new issue, #19446: URL: https://github.com/apache/hudi/issues/19446
### Describe the problem `BaseHoodieCatalystExpressionUtils` recurses into `ParseToDate`/`ParseToTimestamp` via the per-Spark-version `unapplyOrderPreservingDateParsing` hook (#19149) so that `to_date(col)`/`to_timestamp(col)` predicates can be mapped back to the source column for data skipping. Both nodes are `RuntimeReplaceable`. Spark's first optimizer batch (`FinishAnalysis` -> `ReplaceExpressions`) rewrites them before filters are pushed down, so by the time `DataSkippingUtils.translateIntoColumnStatsIndexFilterExpr` sees a predicate from a real query, the `ParseToDate`/`ParseToTimestamp` shapes may no longer exist -- and their replacements (`GetTimestamp`/`Cast` trees) are not in the order-preserving whitelist. If so, `to_date`/`to_timestamp` data skipping is silently a no-op. The existing green coverage is not representative: the `to_timestamp` case in `TestDataSkippingUtils` (around line 677) runs a test-local optimizer that applies only `OptimizeIn`, so the un-replaced node survives there but not in a real query plan. ### Action Add an optimizer-realistic test: resolve `to_date(col, fmt) = lit`, run the full `FinishAnalysis` batch, then call `translateIntoColumnStatsIndexFilterExpr` and assert on the result. Either it proves the hook still fires, or it shows the whitelist needs the replaced shapes instead. Found while reviewing #19405, which pins the hook directly with a unit test; this issue tracks the end-to-end reachability question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
