yaooqinn opened a new pull request, #54072: URL: https://github.com/apache/spark/pull/54072
### What changes were proposed in this pull request? The SQLQueryTestSuite test `in-set-operations.sql` intermittently fails on GitHub Actions CI with `SparkOutOfMemoryError` (`UNABLE_TO_ACQUIRE_MEMORY`). **Root Cause:** The test runs complex queries with multiple UNIONs and correlated IN-subqueries. The physical plan contains 5 BroadcastHashJoin operations with nested BroadcastExchange, which accumulates hash tables in memory. On memory-constrained CI runners (~7GB RAM, 4GB JVM heap), this causes OOM under memory pressure. **Fix:** Add `--SET spark.sql.autoBroadcastJoinThreshold=-1` to the test file to disable broadcast joins. This forces SortMergeJoin which can spill to disk if needed, reducing peak memory usage. ### Why are the changes needed? To fix a flaky test that intermittently fails on CI due to memory pressure. ### Does this PR introduce _any_ user-facing change? No. This only affects test configuration. ### How was this patch tested? - Ran `./build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z in-set-operations.sql"` - all tests passed - The golden file is updated for minor row reordering where ORDER BY keys are equal (non-deterministic ordering for rows with identical sort keys) ### Was this patch authored or co-authored using generative AI tooling? Yes, GitHub Copilot CLI assisted with analysis and implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
