viirya commented on code in PR #7357:
URL: https://github.com/apache/datafusion-comet/pull/7357#discussion_r4646124377
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -1168,52 +1191,33 @@ case class CometExecRule(session: SparkSession,
queryStagePrep: Boolean = false)
// Set up logical links
newPlan = newPlan.transform {
- case op: CometExec
- if op
- .getTagValue(SparkPlan.LOGICAL_PLAN_TAG)
- .exists(_.isInstanceOf[LogicalQueryStage]) =>
- // AQE replanning reuses this physical root and links it to the
current logical stage.
- // originalPlan can still point to a subtree hidden inside that
logical leaf, which
- // AQE cannot replace in the current logical plan. Only preserve a
direct stage link,
- // not a link inherited from an ancestor.
- // On the ordinary exchange path, the exchange itself is behind a
QueryStageExec
- // leaf and is not visited by this transform.
- // Spark 4.1.3 returns the existing root in
LogicalQueryStageStrategy and then calls
- // setLogicalLink from SparkStrategies.plan:
+ case op: CometExec =>
+ // Spark reuses the physical root and links it directly to the
current AQE stage.
+ // A descendant can inherit that stage if it wraps the descendant's
original node.
+ // AQE replaces the linked node by identity. A stale link leaves the
plan unchanged.
// scalastyle:off line.size.limit
//
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala#L64-L65
//
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala#L78-L87
+ //
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L778-L783
// scalastyle:on line.size.limit
- op
- case op: CometExec =>
- if (op.originalPlan.logicalLink.isEmpty) {
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)
- } else {
- op.originalPlan.logicalLink.foreach(op.setLogicalLink)
+ val keepStage = op.logicalLink.exists {
+ case stage: LogicalQueryStage =>
+ // logicalLink prefers the direct tag, so if present it points
to this stage.
+ op.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).isDefined ||
+ op.originalPlan.logicalLink.exists(stageWraps(stage, _))
+ case _ => false
+ }
+ if (!keepStage) {
+ copyLogicalLinks(op.originalPlan, op, clearMissing = true)
}
op
case op: CometShuffleExchangeExec =>
Review Comment:
Does the inherited tag here change any AQE behavior?
`setLogicalLinkForNewQueryStage` reads `logicalLink`, so the stage gets the
same `Sort` either way. And without a `keepStage` check on exchanges, a stage
link Spark later propagates onto a pending exchange is reset on the next pass.
If the goal is only tag-kind parity with Spark, could the comment say that?
"Clear it" also no longer matches what the branch does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]