xumingming opened a new pull request, #57901: URL: https://github.com/apache/spark/pull/57901
### What changes were proposed in this pull request? Fix `PushdownPredicatesAndPruneColumnsForCTEDef` discarding filters that other optimizer rules injected into a CTE definition between this rule's applications. The rule previously rebuilt a CTE definition from the frozen pre-push-down snapshot stored in `CTERelationDef.originalPlanWithPredicates` whenever newly gathered reference predicates re-armed its guard. With this PR the rule rebuilds from the **current** child instead: it removes only the push-down filter it placed in the previous pass (located by a descent that mirrors the predicate push-down rules' own translations through `Project`/`Aggregate` aliases — using the same `AliasHelper` utilities so the two cannot drift — positionally into `Union` branches, and unchanged through `Join`/`Window`/output-preserving unary nodes), then wraps with the latest combined predicate. If the previous push can no longer be located (another rule rewrote or merged it), the current child is used as-is: re-pushing the disjunction of reference predicates is redundant but always semantics-preserving, since every reference re-applies its own predicates. A cross-reference note is added to `PushPredicateThroughNonJoin` so future changes to push-through cases update the mirror. ### Why are the changes needed? The rule runs in both fixedPoint operator-optimization batches, with the `Once` batch "Infer Filters" (`InferFiltersFromConstraints`) sandwiched between them. Pass 1 snapshots the pre-push-down child; the `Once` batch then injects new filters into the definition's child (e.g. propagating a pushed predicate through a join condition to the other side) and enriches reference-site predicates (e.g. adding `IsNotNull`). The enrichment re-arms the rule's guard in pass 2, and the snapshot rebuild permanently discards the injected filters — losing partition pruning / data source filter push-down. SPARK-58696 has a minimal repro on stock Spark (t2's `Range` ends up with no filter while t1's keeps `((id = 5) OR (id = 7))`) and a production incident where a missing inferred partition filter turned a pruned read into a full scan. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? - New `PushdownPredicatesForCTEDefStalenessSuite` (6 tests): the staleness regression test **fails on unmodified master**; further tests cover idempotency under foreign mutation and previous-push removal through `Project`/`Join`/`Union`/`Aggregate`/`Window`. - `CTEInlineSuite*` + `CTEHintSuite`: 63/63 pass — the plan shapes asserted by existing CTE push-down tests (including "combined predicate") are unchanged. - Full `sql/catalyst` optimizer package: 1401/1401 pass. - `catalyst/scalastyle` and `catalyst/Test/scalastyle`: clean. ### Was this patch authored or generated with the assistance of AI tools? Yes. An AI coding agent (Claude) assisted with root-cause analysis, reproduction, fix design, and test authoring. All changes were reviewed, directed, and validated by the author; full test suites were run locally as listed above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
