Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/20407#discussion_r167392509
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -156,6 +156,15 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val ALLOW_NESTEDJOIN_FALLBACK =
buildConf("spark.sql.join.broadcastJoinFallback.enabled")
+ .internal()
+ .doc("When true (default), if the other options are not available,
fallback to try and use " +
+ "BroadcastNestedLoopJoin as join strategy. This can cause OOM which
can be a problem " +
+ "in some scenarios, eg. when running the thriftserver. Turn to false
to disable it: an " +
+ "AnalysisException will be thrown.")
--- End diff --
OOM could be triggered by various cases. `BroadcastNestedLoopJoin` is being
widely used in many cases. If this is the case in your scenario, you can do it
in your fork.
Also, this PR just helps a very specific case. I do not think it resolves
the general issue.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
