Github user maropu commented on a diff in the pull request:
https://github.com/apache/spark/pull/21754#discussion_r203966689
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/Exchange.scala
---
@@ -85,14 +85,20 @@ case class ReusedExchangeExec(override val output:
Seq[Attribute], child: Exchan
*/
case class ReuseExchange(conf: SQLConf) extends Rule[SparkPlan] {
+ private def supportReuseExchange(exchange: Exchange): Boolean = exchange
match {
+ // If a coordinator defined in an exchange operator, the exchange
cannot be reused
--- End diff --
We might be able to logically reuse the same coordinator though, it seems
to be difficult to implement based on the current master, I think. In the
current adaptive query execution, exchanges (between stages) registered in a
coordinator and their partition size are decided on runtime (inside
`SparkPlan.execute()`). Since `ReuseExchange` runs in the final phase of
planning. So, it is difficult to tell which coordinator can be reused at that
time. So, to archive the reuse, we might need some refactoring about these
logics...
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
