voonhous opened a new issue, #19479: URL: https://github.com/apache/hudi/issues/19479
# Title [SUPPORT] Data skipping: left(col, n) predicates never prune -- whitelist arm matches a shape the optimizer eliminated (SPARK-38240) # Body **Describe the problem you faced** Same bug class as #19446, different expression. `Left` has been RuntimeReplaceable in every Spark version Hudi supports (3.3 through 4.2): the optimizer's ReplaceExpressions rewrites `left(str, n)` into `Substring(str, 1, n)` before any filter reaches Hudi's file pruning. The order-preserving whitelist in `BaseHoodieCatalystExpressionUtils.OrderPreservingTransformation` only matches the pre-replacement `Left` shape, so `left(col, n) = '...'` predicates translate to TrueLiteral and data skipping silently prunes nothing. The arm even sits under a comment citing SPARK-38240 -- the arity of a node that never arrives was kept up to date across Spark upgrades. **Evidence** - `javap -cp spark-catalyst_2.12-3.3.4.jar org.apache.spark.sql.catalyst.expressions.Left` -> `implements RuntimeReplaceable` (same for 3.4.3, 3.5.6, 4.0.2, 4.1.1, 4.2.0); replacement is `Substring(str, Literal(1), len)` (visible in bytecode). - Live optimizer probe: `left(B, 4) > '2021'` optimizes to `substring(B#0, 1, 4)`, which the whitelist does not match (`lower(B)` by contrast still matches). - Archaeology: the `Left` arm was added together with `ParseToDate`/`ParseToTimestamp` in HUDI-3594 (#4996, Mar 2022); all three were already RuntimeReplaceable in Spark 3.2, so the arm has never fired on the read path. **Suggested fix** Add a prefix-only Substring arm next to the existing string arms: ```scala case Substring(OrderPreservingTransformation(attrRef), Literal(1, _), _) => Some(attrRef) ``` (a non-1 `pos` is NOT order-preserving), delete the dead `Left` arm, and add a `left(B, 4) > '...'` row to the full-optimizer method source introduced in #19474. Note prefix-substring over the string min/max stats is order-preserving and total, so it needs neither the format gate nor extra null handling. **Environment**: master (found during review of #19474). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
