LukaZdravic commented on code in PR #59957:
URL: https://github.com/apache/spark/pull/59957#discussion_r4455876882
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:
##########
@@ -434,6 +464,63 @@ private[joins] class SortMergeAsOfJoinScanner(
null
}
+ /**
+ * Finds the best right row for `leftRow` by moving a cursor over the right
input, or null.
+ *
+ * Left rows of a group come in ascending as-of key order. So a right row
that passes for one
+ * left row also passes for the next (backward), and a row that fails stays
failed (forward).
+ * The cursor never steps back: backward stops at the first failing row and
keeps it for the
+ * next left row, and forward drops each failing row, since no later left
row can match it.
+ */
+ private def findWithCursor(leftRow: InternalRow, leftKey: UnsafeRow):
InternalRow = {
+ equiKeyOrdering match {
+ case Some(ordering)
+ if cursorGroupKey == null || ordering.compare(leftKey,
cursorGroupKey) != 0 =>
+ cursorGroupKey = leftKey
+ cursorBest = null
+ case _ =>
+ }
+ val (leftAsOfKey, rightAsOfKey) = cursorKeys
+ // A NULL left as-of key matches nothing. It must not move the cursor, or
forward would drop
+ // right rows that later left rows need.
+ if (leftAsOfKey.eval(leftRow) == null) return null
Review Comment:
Fixed, guarded against it with checking if the right side has anything to
match the left side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
