cloud-fan commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3489126398
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala:
##########
@@ -814,7 +814,11 @@ case class AdaptiveSparkPlanExec(
try {
logicalPlan.invalidateStatsCache()
val optimized = optimizer.execute(logicalPlan)
- val sparkPlan =
context.session.sessionState.planner.plan(ReturnAnswer(optimized)).next()
+ // Go through QueryExecution.createSparkPlan -- the single place that
strips DelegateExpression
+ // before planning -- instead of calling the planner directly, so the
re-planned stage sees the
+ // real executed expression.
+ val sparkPlan = QueryExecution.createSparkPlan(
Review Comment:
Fixed in bffecc826d6 — `reOptimize` now lowers once and returns the same
lowered tree it planned, so the physical plan's `logicalLink` targets stay
reference-consistent with the returned logical plan.
On the regression test: I couldn't find a way to assert this one
deterministically. The symptom is silent — no wrong results and no error
(eval/codegen still delegate), and the physical `logicalLink`s are identical
with or without the fix (they're set on the lowered copy in `createSparkPlan`
either way). The only thing that differs is `currentLogicalPlan`, a transient
local in `getFinalPhysicalPlan`, whose orphaned-stage state surfaces only as a
threshold-sensitive re-planning *decision* on a later iteration. A test would
need either a flaky multi-stage stats-driven plan-difference query or test-only
instrumentation to expose stage tracking. I verified the fix by reasoning
through `replaceWithQueryStagesInLogicalPlan` plus a full
`AdaptiveQueryExecSuite` run (green). Open to adding observability
instrumentation in a follow-up if you think it's warranted — let me know if
you'd prefer that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
