uros-b commented on code in PR #56595:
URL: https://github.com/apache/spark/pull/56595#discussion_r3443048484
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/SortMergeAsOfJoinExec.scala:
##########
@@ -429,27 +403,23 @@ private[joins] class SortMergeAsOfJoinScanner(
if (residualSatisfied) {
val distance = boundOrderExpr.eval(joinedRow)
if (distance != null) {
- if (bestMatch == null) {
- bestMatch = rightRow
- bestDistance = distance
- } else if (distanceOrdering.lt(distance, bestDistance)) {
- bestMatch = rightRow
+ if (bestMatch == null || distanceOrdering.lt(distance,
bestDistance)) {
+ bestMatch = rightRow.copy()
bestDistance = distance
} else {
- // Distance is increasing; for Forward the as-of condition
- // guarantees no closer row exists further right. For
- // Nearest the distance is V-shaped so once past the
- // minimum no later row can beat it.
+ // Distance is increasing past the minimum. For Forward,
+ // the as-of condition guarantees no closer row exists
+ // further right. For Nearest, distance is V-shaped so
+ // once past the minimum no later row can beat it.
return bestMatch
}
}
}
}
- // Note: we do NOT early-terminate on as-of condition failure here.
+ // Do NOT early-terminate on as-of condition failure here.
// For Nearest + !allowExactMatches, the condition is false at a
// single interior point (right == left) with valid matches on
- // both sides. The distance-based termination above is sufficient.
- i += 1
+ // both sides. Distance-based termination above is sufficient.
}
Review Comment:
Do we have any test coverage? The 26 existing SortMergeAsOfJoinSuite tests
cover join semantics, but all stay in-memory. The benchmark uses only 100
groups with no spill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
