cloud-fan commented on PR #58870:
URL: https://github.com/apache/spark/pull/58870#issuecomment-5798314564
Thanks for the detailed review. I revisited the scope and simplified the
patch around the original
invariant: the applicable automatic broadcast threshold is a floor for the
dedicated NAAJ
threshold.
The implementation is now the direct predicate: a negative dedicated
threshold, a positive
dedicated threshold that admits the right side, or `canBroadcastBySize`
admitting the right side.
The hint/build-side emulation and the configuration-only statistics short
circuit are removed.
The intended assumptions are now explicit in the code, configuration
documentation, tests, and PR
description:
* Threshold eligibility precedes join hints, matching the behavior of the
dedicated threshold
before this PR. Hints affect the nested-loop fallback only after threshold
eligibility rejects
the specialized NAAJ. This also removes the partial
`NO_BROADCAST_AND_REPLICATION` handling.
* The shared eligibility decision intentionally continues to control
aggregate pushdown. The
optimizer makes that decision from estimated statistics; AQE may later
reevaluate physical join
selection with runtime statistics. That estimated-versus-runtime risk
already exists for a
positive dedicated threshold and is not a correctness issue.
* This PR floors the dedicated threshold by the automatic threshold. The
final forced BNLJ path
that broadcasts the right side above both thresholds is pre-existing and
outside this narrower
contract.
I also updated the stale method Scaladoc and shortened the configuration
documentation, switched
the end-to-end cases to `assertJoin`, explicitly set the NAAJ optimization
flag in the pushdown
test, added coverage for hints not overriding thresholds and for `BigInt`
sizes, and restored both
SPARK-36082 and SPARK-59673 in the regression-test name so the documented
test command selects it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
