cloud-fan commented on code in PR #56575:
URL: https://github.com/apache/spark/pull/56575#discussion_r3449750709
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/NormalizePlan.scala:
##########
@@ -78,8 +77,23 @@ object NormalizePlan extends PredicateHelper {
* [[InheritAnalysisRules]] is the replacement expression, the original
expression will be lost
* and timezone will never be applied. This causes inconsistencies, because
fixed-point semantic
* is to ALWAYS apply timezone, regardless of whether the Cast actually
needs it.
+ *
+ * Note: this unconditionally unfolds every [[RuntimeReplaceable]] into its
`replacement`. It
+ * intentionally does NOT reuse the `ReplaceExpressions` optimizer rule,
which now keeps
+ * deterministic, evaluable [[RuntimeReplaceable]] nodes in the plan (they
are materialized later,
+ * before codegen). Normalization still needs the fully-unfolded form so
that the non-child
+ * `parameters` of [[InheritAnalysisRules]] (e.g. the original, un-timezoned
cast) are dropped.
*/
- def normalizeRuntimeReplaceable(plan: LogicalPlan): LogicalPlan =
ReplaceExpressions(plan)
+ def normalizeRuntimeReplaceable(plan: LogicalPlan): LogicalPlan = {
+ plan.transformWithPruning(_.containsAnyPattern(RUNTIME_REPLACEABLE)) {
+ case p => p.mapExpressions(replaceRuntimeReplaceable)
+ }
+ }
+
+ private def replaceRuntimeReplaceable(e: Expression): Expression = e match {
+ case r: RuntimeReplaceable => replaceRuntimeReplaceable(r.replacement)
Review Comment:
Done in b80438b. Extracted a shared `RuntimeReplaceable.unfold` helper
(companion object in `Expression.scala`); both
`NormalizePlan.normalizeRuntimeReplaceable` and
`MaterializeRuntimeReplaceable.apply` now call it, removing the two private
copies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
