Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/20678#discussion_r170760936
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -1068,6 +1068,14 @@ object SQLConf {
.booleanConf
.createWithDefault(false)
+ val ARROW_FALLBACK_ENABLE =
+ buildConf("spark.sql.execution.arrow.fallback.enabled")
+ .doc("When true, the optimization by
'spark.sql.execution.arrow.enabled' " +
+ "could be disabled when it is unable to be used, and fallback to "
+
+ "non-optimization.")
--- End diff --
Just a suggestion: "When true, optimizations enabled by
'spark.sql.execution.arrow.enabled' will fallback automatically to
non-optimized implementations if an error occurs."
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
