c21 commented on a change in pull request #29455:
URL: https://github.com/apache/spark/pull/29455#discussion_r472583133
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala
##########
@@ -235,8 +235,13 @@ abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
.getOrElse(createJoinWithoutHint())
case j @ ExtractSingleColumnNullAwareAntiJoin(leftKeys, rightKeys) =>
- Seq(joins.BroadcastHashJoinExec(leftKeys, rightKeys, LeftAnti,
BuildRight,
- None, planLater(j.left), planLater(j.right), isNullAwareAntiJoin =
true))
+ if (!canBroadcastBySize(j.right, conf) &&
conf.adaptiveExecutionEnabled) {
Review comment:
Sorry I don't get why we don't worry about shuffled hash join OOM if
sub-query is way too big/skewed in certain keys. In general, shuffled hash join
OOM is a normal issue if people choose shuffled hash join blindly. Could we add
a check to make sure build size is not too big to be shuffled hash join?
e.g.
```
if (!canBroadcastBySize(j.right, conf) &&
canBuildLocalHashMapBySize(j.right, conf) && conf.adaptiveExecutionEnabled) {
...
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
