cloud-fan commented on code in PR #58419:
URL: https://github.com/apache/spark/pull/58419#discussion_r4000523239
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala:
##########
@@ -1059,12 +1116,31 @@ case class UnionExec(children: Seq[SparkPlan]) extends
SparkPlan with CodegenSup
}
}
- // Memoized: consulted by `supportCodegen` (called multiple times by
- // `CollapseCodegenStages`) and by `metrics`. Conf and children are stable
- // for a given UnionExec instance; cross-plan staleness is impossible since
- // UnionExec is a case class and `withNewChildren` produces a fresh instance.
+ // The confs the gate reads, stamped for the reason the plain-union decision
is: `conf` is live,
+ // so the gate, `metrics` and the copy `insertInputAdapter` puts inside the
codegen shell would
+ // otherwise be free to read different values. When a child is not
`CodegenSupport` that copy is
+ // real and its first evaluation lands at execution; reading the conf there
left `metrics` empty
+ // while `doProduce` asked `metricTerm` for `numOutputRows`. A read before
the stamp answers from
+ // the conf as it is then and writes nothing, so observing an unprepared
plan cannot pin this
+ // either.
+ private def unionCodegenEnabled: Boolean =
+ stampedDecisions.map(_.unionCodegenEnabled)
+ .getOrElse(conf.getConf(SQLConf.WHOLESTAGE_UNION_CODEGEN_ENABLED))
+
+ private def maxCodegenChildren: Int =
+ stampedDecisions.map(_.maxChildren)
+ .getOrElse(conf.getConf(SQLConf.WHOLESTAGE_UNION_MAX_CHILDREN))
+
+ // Memoized per instance rather than stamped on the tag. Every term below
the confs except
+ // `isPlainUnion` reads the children, and a tag outlives them:
`SparkPlanInfo` forces `metrics` on
Review Comment:
Confirmed: the comment now uses the initial `SQLExecution` `SparkPlanInfo`
metrics read as the trigger and no longer relies on the reversed event order.
Resolved.
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"inline:3996975952","thread_id":"inline:3996975952","verdict_sha256":"3f145606514841e98aa4f828110c8e9a2ef2a823fe82b8c1858abbafd71ce3c8"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
