sririshindra commented on a change in pull request #31477:
URL: https://github.com/apache/spark/pull/31477#discussion_r570672502
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala
##########
@@ -89,13 +90,20 @@ case class BroadcastNestedLoopJoinExec(
}
}
- @transient private lazy val boundCondition = {
+ private val numMatchedPairs = longMetric("numMatchedPairs")
+
+ @transient private lazy val boundCondition: InternalRow => Boolean =
if (condition.isDefined) {
Review comment:
If the condition is empty, then `numOutputRows == numMatchedPairs` only
in the case of InnerJoin. It is not necessarily the case for other types of
join, because numMatchedPairs is only measuring the records when there is a
common key on both the build side and the stream side. For instance, in the
case of leftOuterJoin `numOutputRows = numMatchedPairs + (number of non
matched pairs on the left side of the join)`.
Moreover, if we define a metric and fail to populate it for some cases, then
the existing unit tests in the SQLMetricsSuite will fail because of the way the
unit tests fetch the actual metrics.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
