cloud-fan commented on code in PR #58870:
URL: https://github.com/apache/spark/pull/58870#discussion_r4089663360
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -438,12 +437,18 @@ trait JoinSelectionHelper extends Logging {
getBroadcastBuildSide(join, hintOnly = true, conf).orElse {
if (noShufflePlannedBefore) getBroadcastBuildSide(join, hintOnly =
false, conf) else None
}
- // `JoinSelection` always builds from the right for this shape. A negative
threshold preserves
- // the original unbounded NAAJ behavior, while zero disables the broadcast
hash optimization.
+ // `JoinSelection` always builds from the right for this shape. The
applicable automatic
+ // broadcast threshold floors a nonnegative dedicated threshold. As
before, threshold
+ // eligibility takes precedence over join hints. This same decision
intentionally controls
+ // aggregate pushdown.
case j @ ExtractSingleColumnNullAwareAntiJoin(_, _) =>
- val threshold = conf.nullAwareAntiJoinBroadcastThreshold
- val rightSize = j.right.stats.sizeInBytes
- if (threshold < 0 || (threshold > 0 && rightSize >= 0 && rightSize <=
threshold)) {
+ val dedicatedThreshold = conf.nullAwareAntiJoinBroadcastThreshold
+ val canBroadcast = dedicatedThreshold < 0 ||
+ (dedicatedThreshold > 0 && {
+ val rightSize = j.right.stats.sizeInBytes
+ rightSize >= 0 && rightSize <= dedicatedThreshold
+ }) || canBroadcastBySize(j.right, conf)
Review Comment:
Agreed that the two decisions can diverge. With an explicitly nonnegative
dedicated threshold, the main optimizer may push the join below the aggregate
using estimated size and the static threshold; AQE can later reject the hash
join under a lower adaptive threshold without reversing that pushdown. Commit
`7c4829dbb42` adds this consequence to the config documentation. The dedicated
threshold's default remains `-1` (unbounded), so this change in pushdown
eligibility requires an explicit nonnegative setting.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -438,12 +437,18 @@ trait JoinSelectionHelper extends Logging {
getBroadcastBuildSide(join, hintOnly = true, conf).orElse {
if (noShufflePlannedBefore) getBroadcastBuildSide(join, hintOnly =
false, conf) else None
}
- // `JoinSelection` always builds from the right for this shape. A negative
threshold preserves
- // the original unbounded NAAJ behavior, while zero disables the broadcast
hash optimization.
+ // `JoinSelection` always builds from the right for this shape. The
applicable automatic
+ // broadcast threshold floors a nonnegative dedicated threshold. As
before, threshold
+ // eligibility takes precedence over join hints. This same decision
intentionally controls
+ // aggregate pushdown.
Review Comment:
Agreed. For an unhinted LeftAnti, a negative automatic threshold can still
reach the last-resort BuildRight nested-loop join. Commit `7c4829dbb42`
clarifies beside the predicate that the thresholds limit hash relation
construction, not all right-side broadcasts. We keep the size gate because
broadcasting rows for a nested-loop join and building a hash relation have
different memory costs, and a nonnegative dedicated threshold bounds the latter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
