sunchao commented on code in PR #7357:
URL: https://github.com/apache/datafusion-comet/pull/7357#discussion_r4637456786
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -1108,37 +1131,29 @@ case class CometExecRule(session: SparkSession,
queryStagePrep: Boolean = false)
// Set up logical links
newPlan = newPlan.transform {
- case op: CometExec
- if op
- .getTagValue(SparkPlan.LOGICAL_PLAN_TAG)
- .exists(_.isInstanceOf[LogicalQueryStage]) =>
- // AQE replanning reuses this physical root and links it to the
current logical stage.
- // originalPlan can still point to a subtree hidden inside that
logical leaf, which
- // AQE cannot replace in the current logical plan. Only preserve a
direct stage link,
- // not a link inherited from an ancestor.
- // On the ordinary exchange path, the exchange itself is behind a
QueryStageExec
- // leaf and is not visited by this transform.
- // Spark 4.1.3 returns the existing root in
LogicalQueryStageStrategy and then calls
- // setLogicalLink from SparkStrategies.plan:
+ case op: CometExec =>
+ // Spark reuses the physical root and links it directly to the
current AQE stage.
+ // A descendant can inherit that stage if it wraps the descendant's
original node.
+ // AQE replaces the linked node by identity. A stale link leaves the
plan unchanged.
// scalastyle:off line.size.limit
//
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LogicalQueryStageStrategy.scala#L64-L65
//
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala#L78-L87
+ //
https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L778-L783
// scalastyle:on line.size.limit
- op
- case op: CometExec =>
- if (op.originalPlan.logicalLink.isEmpty) {
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
- op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)
- } else {
- op.originalPlan.logicalLink.foreach(op.setLogicalLink)
+ val keepStage = op.logicalLink.exists {
+ case stage: LogicalQueryStage =>
+ // logicalLink prefers the direct tag, so if present it points
to this stage.
+ op.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).isDefined ||
+ op.originalPlan.logicalLink.exists(stageWraps(stage, _))
+ case _ => false
+ }
+ if (!keepStage) {
+ copyLogicalLinks(op.originalPlan, op, clearMissing = true)
}
op
case op: CometShuffleExchangeExec =>
- // Original Spark shuffle exchange operator might have empty logical
link.
- // But the `setLogicalLink` call above on downstream operator of
- // `CometShuffleExchangeExec` will set its logical link to the
downstream
- // operators which cause AQE behavior to be incorrect. So we need to
unset
- // the logical link here.
+ // An exchange without an original logical link can inherit an
unrelated one from
+ // its parent. Clear it so AQE does not associate the exchange with
that parent.
if (op.originalPlan.logicalLink.isEmpty) {
op.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
op.unsetTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG)
Review Comment:
Both exchange branches now use `copyLogicalLinks(op.originalPlan, op,
clearMissing = true)`. I found an inherited-only source in Spark 4.x. For an
ordered `LIMIT` inside a subquery, Spark can insert a local sort above a
single-partition shuffle. Spark tags the sort directly, and the shuffle
inherits the same logical sort. Calling `setLogicalLink` on the converted
shuffle made that tag direct, which stopped future updates from its parent. The
new regression checks that the link stays inherited and points to the exact
same object. [Spark sort
insertion](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/InsertSortForLimitAndOffset.scala#L42-L64)
·
[Regression](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala#L2450-L2466)
AQE reads the exchange’s temporary tag first, then its direct or inherited
link. It searches the children only when the exchange has neither, so repair
can copy the two tags on the exchange itself and let each child keep its own
link. This also covers the empty-link case in #323. The existing DPP test
checks the temporary tag on a reused broadcast. [AQE
lookup](https://github.com/apache/spark/blob/v4.1.3/sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala#L718-L727)
· [DPP
test](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1700-L1733)
I kept the helper scoped to the planner rules. The transition and cache
rules normally receive fresh, untagged wrappers. The scan fallbacks rebuild
leaves, where a residual filter can give the scan an inherited link. Those
construction and fallback paths belong in the broader audit in #6034.
[d9f90a6a8](https://github.com/apache/datafusion-comet/pull/7357/commits/d9f90a6a8eab1ae45584685df58958333b4ccdba)
##########
spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala:
##########
@@ -1429,7 +1429,21 @@ class CometExecRuleSuite extends CometTestBase {
}
}
- test("CometExecRule repairs ordinary and inherited logical links from the
original plan") {
+ test("CometExecRule preserves an inherited AQE stage for its original
logical subtree") {
+ withAdaptiveAggregateConf {
+ val aggregate = createAdaptiveAggregate()
+ val original = aggregate.originalPlan.logicalLink.get
+ val stage = LogicalQueryStage(LogicalQueryStage(original, aggregate),
aggregate)
+ aggregate.unsetTagValue(SparkPlan.LOGICAL_PLAN_TAG)
+ aggregate.setTagValue(SparkPlan.LOGICAL_PLAN_INHERITED_TAG, stage)
+
+ val repaired = applyCometExecRule(aggregate)
+ assert(repaired.getTagValue(SparkPlan.LOGICAL_PLAN_TAG).isEmpty)
+ assert(repaired.logicalLink.exists(_ eq stage))
+ }
+ }
Review Comment:
The three tests now share the aggregate setup and run with a direct,
inherited, or absent tag on the original. The nested-stage test checks that the
exact outer stage survives as an inherited link in both tagged cases. With an
untagged original, it checks that repair clears the link. [Shared
setup](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1313-L1331)
· [Nested-stage
assertions](https://github.com/apache/datafusion-comet/blob/d9f90a6a8eab1ae45584685df58958333b4ccdba/spark/src/test/scala/org/apache/comet/rules/CometExecRuleSuite.scala#L1436-L1450)
·
[d9f90a6a8](https://github.com/apache/datafusion-comet/pull/7357/commits/d9f90a6a8eab1ae45584685df58958333b4ccdba)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]