cloud-fan commented on PR #56575: URL: https://github.com/apache/spark/pull/56575#issuecomment-4814080427
@sunchao Heads-up: I've reworked this PR substantially and force-pushed, so the existing review threads now point at code that has been removed. Thank you for the earlier reviews — they're what drove the redesign. Instead of letting `RuntimeReplaceable` opt out of eager replacement and *survive* into the physical plan (with `MaterializeRuntimeReplaceable` and per-consumer unfolds), the PR now adds `DelegateExpression`: a transparent, named **logical-plan** wrapper whose `definition` is a real child, fully visible to the analyzer and optimizer. `RuntimeReplaceable` is reverted to exactly master. The wrapper is stripped to its `definition` at the single entry to the planner (`QueryExecution.createSparkPlan`, which AQE re-planning also routes through), so the planner and every physical consumer — join-key extraction, pushdown, columnar, codegen — operate on the real executed expression. V2 pushdown is a logical rule (it runs before the strip), so it unfolds the wrapper directly in `V2ExpressionBuilder`. The intent is to dissolve the class of issues your `[P2]`s identified — the lowered form hidden from finish-analysis rules and from planner-level structural matchers — at the root (keep the definition a visible child, and lower it before planning) rather than patching each consumer. The migrated consumers (`right`, `multi_get_json_object`) are scalar/value functions. The updated PR description has the full design, including one documented limitation (a delegate created by a *physical* rule isn't stripped and may reach an external system un-lowered — acceptable, like any expression it doesn't recognize). Would appreciate a fresh look when you have time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
