cloud-fan opened a new pull request, #56575: URL: https://github.com/apache/spark/pull/56575
### What changes were proposed in this pull request? Today every `RuntimeReplaceable` is rewritten into its `replacement` by `ReplaceExpressions` in the logical optimizer, so it never reaches the physical plan. This PR lets an evaluable `RuntimeReplaceable` *survive* into the physical plan: - `ReplaceExpressions` now rewrites early **only** when the fully-expanded replacement contains an `Unevaluable` expression (e.g. `With`, which depends on the logical-phase `RewriteWithExpression` rule); `RuntimeReplaceableAggregate` is still always rewritten. Everything else survives. - A new physical-preparation rule `MaterializeRuntimeReplaceable` rewrites the survivors into their replacement after columnar/native conversion and before `CollapseCodegenStages`, so Spark whole-stage codegen never sees a `RuntimeReplaceable` and per-operator metrics stay intact. - `RuntimeReplaceable.eval`/`doGenCode` delegate to `replacement` as a backstop for interpreted and non-whole-stage-codegen paths. ### Why are the changes needed? Keeping the semantic expression (e.g. `right(a, b)`) in the plan lets optimizer rules introduce `RuntimeReplaceable`s freely, keeps the high-level expression visible in the optimized logical plan, and lets a native engine match the high-level expression directly instead of reverse-engineering its lowered form. ### Does this PR introduce _any_ user-facing change? `EXPLAIN` of the optimized logical plan now shows the original expression (e.g. `nvl`) rather than its replacement (e.g. `coalesce`) for the surviving cases. The physical plan and query results are unchanged. ### How was this patch tested? `StringFunctionsSuite` and `JsonFunctionsSuite` pass locally; relying on full CI to surface any plan-shape or execution regressions from `RuntimeReplaceable` surviving the optimizer. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
