hvanhovell commented on a change in pull request #25167: [SPARK-27485]
EnsureRequirements.reorder should handle duplicate expressions gracefully
URL: https://github.com/apache/spark/pull/25167#discussion_r303669780
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala
##########
@@ -117,25 +116,41 @@ case class EnsureRequirements(conf: SQLConf) extends
Rule[SparkPlan] {
}
private def reorder(
- leftKeys: Seq[Expression],
- rightKeys: Seq[Expression],
+ leftKeys: IndexedSeq[Expression],
+ rightKeys: IndexedSeq[Expression],
expectedOrderOfKeys: Seq[Expression],
currentOrderOfKeys: Seq[Expression]): (Seq[Expression], Seq[Expression])
= {
- val leftKeysBuffer = ArrayBuffer[Expression]()
- val rightKeysBuffer = ArrayBuffer[Expression]()
- val pickedIndexes = mutable.Set[Int]()
- val keysAndIndexes = currentOrderOfKeys.zipWithIndex
-
- expectedOrderOfKeys.foreach(expression => {
- val index = keysAndIndexes.find { case (e, idx) =>
- // As we may have the same key used many times, we need to filter out
its occurrence we
- // have already used.
- e.semanticEquals(expression) && !pickedIndexes.contains(idx)
- }.map(_._2).get
- pickedIndexes += index
- leftKeysBuffer.append(leftKeys(index))
- rightKeysBuffer.append(rightKeys(index))
- })
+ if (expectedOrderOfKeys.size != currentOrderOfKeys.size) {
+ return (leftKeys, rightKeys)
+ }
+
+ // Build a lookup between an expression and the positions its holds in the
current key seq.
Review comment:
The previous implementation had the potential for quadratic behavior in
quite a few places so I changed all that. I might have gotten carried away
here, especially given the fact that number of keys is often quite low.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
