ahshahid commented on code in PR #43806:
URL: https://github.com/apache/spark/pull/43806#discussion_r1393590377
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SubqueryAdaptiveBroadcastExec.scala:
##########
@@ -44,9 +46,21 @@ case class SubqueryAdaptiveBroadcastExec(
throw
QueryExecutionErrors.executeCodePathUnsupportedError("SubqueryAdaptiveBroadcastExec")
}
+ override def equals(other: Any): Boolean = other match {
+ case x: SubqueryAdaptiveBroadcastExec => this.name == x.name && this.index
== x.index &&
+ this.onlyInBroadcast == x.onlyInBroadcast && this.buildPlan ==
x.buildPlan &&
+ this.buildKeys == x.buildKeys && this.child == x.child
+ case y: SubqueryBroadcastExec => this.name == y.name && this.index ==
y.index &&
+ this.buildKeys == y.buildKeys && this.child == y.child
Review Comment:
I don't think that would work as such and may actually complicate the
things. the issue lies in the following code :
In AdaptivePlanExec:
where we look for exising stage for reuse:
`
if (conf.exchangeReuseEnabled) {
// Check the `stageCache` again for reuse. If a match is
found, ditch the new stage
// and reuse the existing stage found in the `stageCache`,
otherwise update the
// `stageCache` with the new stage.
val queryStage = context.stageCache.getOrElseUpdate(
newStage.plan.canonicalized, newStage)
if (queryStage.ne(newStage)) {
newStage = reuseQueryStage(queryStage, e)
}
}
`
The canonicalized plan contains SubqueryAdaptiveBroadcastExec, while
incoming canonicalized plan contains SubqueryBroadcastExec. and hence mistmatch.
Any replacement , will potentially complicate things and cause perf issue
too, because these nodes though extending SparkPlan, are actually embedded in
ExecSubqueryExpression which means checking for each expression in all the
spark plan nodes to modify. and then potentially adjusting the logical plan
tags too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
