cloud-fan opened a new pull request, #59091: URL: https://github.com/apache/spark/pull/59091
### What changes were proposed in this pull request? This follow-up restores the three-way contract for `spark.sql.optimizer.nullAwareAntiJoinBroadcastThreshold`: - A negative value allows the specialized null-aware anti join broadcast hash path without a size limit. - Zero always disables that hash path. - A positive value admits the path when either the dedicated threshold or the applicable automatic broadcast threshold allows it. It updates the shared join-selection logic, configuration documentation, and optimizer and execution tests. ### Why are the changes needed? SPARK-36082 established zero as the sentinel for disabling broadcast NAAJ. SPARK-59673 made the automatic broadcast threshold a floor, but unintentionally allowed that threshold to override zero. This restores the existing disable sentinel while preserving the automatic-threshold floor for positive dedicated thresholds. ### Does this PR introduce _any_ user-facing change? Yes. On current master, setting `spark.sql.optimizer.nullAwareAntiJoinBroadcastThreshold` to zero once again disables the specialized NAAJ broadcast hash path even when an automatic broadcast threshold is positive. This corrects behavior introduced by the unreleased SPARK-59673 change. ### How was this patch tested? Updated `JoinSelectionHelperSuite`, `LeftSemiAntiJoinPushDownSuite`, and `JoinSuite` to cover the zero disable sentinel and the positive-threshold behavior. The following focused suites were attempted: - `catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.JoinSelectionHelperSuite` - `catalyst/testOnly org.apache.spark.sql.catalyst.optimizer.LeftSemiAntiJoinPushDownSuite` - `sql/testOnly org.apache.spark.sql.JoinSuite -- -z SPARK-36082` The build could not reach compilation because the environment could not resolve `repo.maven.apache.org` while fetching Maven dependencies. `git diff --check`, changed-file line-length checks, and changed-file non-ASCII checks passed. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: OpenAI Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
