agrawaldevesh commented on a change in pull request #29104:
URL: https://github.com/apache/spark/pull/29104#discussion_r457585404
##########
File path:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/planning/patterns.scala
##########
@@ -388,3 +390,37 @@ object PhysicalWindow {
case _ => None
}
}
+
+object ExtractSingleColumnNullAwareAntiJoin extends JoinSelectionHelper {
+
+ // SingleColumn NullAwareAntiJoin
+ // streamedSideKeys, buildSideKeys
+ // currently these two return Seq[Expression] should have only one element
+ private type ReturnType =
Review comment:
nit: Put this on the same line
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastHashJoinExec.scala
##########
@@ -64,10 +65,32 @@ case class BroadcastHashJoinExec(
val numOutputRows = longMetric("numOutputRows")
val broadcastRelation = buildPlan.executeBroadcast[HashedRelation]()
- streamedPlan.execute().mapPartitions { streamedIter =>
- val hashed = broadcastRelation.value.asReadOnlyCopy()
-
TaskContext.get().taskMetrics().incPeakExecutionMemory(hashed.estimatedSize)
- join(streamedIter, hashed, numOutputRows)
+ if (isNullAwareAntiJoin) {
+ streamedPlan.execute().mapPartitionsInternal { streamedIter =>
+ if (broadcastRelation.value.inputEmpty) {
+ streamedIter
+ } else if (broadcastRelation.value.anyNullKeyExists) {
Review comment:
Yeah, NAAJ is super super crazy and I think @leanken's logic is correct.
It cannot be generalized easily to the "distributed" case of shuffled hash
join, so let's ONLY fix this for broadcast joins -- This also maintains parity
with existing broadcast nested loop join that will invariably do a broadcast
only join for NAAJ. (So it's not like we have increased the memory needed by
this PR)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
