ueshin commented on code in PR #46553:
URL: https://github.com/apache/spark/pull/46553#discussion_r1598873783
##########
python/pyspark/pandas/utils.py:
##########
@@ -492,6 +504,10 @@ def default_session() -> SparkSession:
"from pandas API on Spark since pandas API on Spark follows "
"the behavior of pandas, not SQL."
)
+ default_mode_value = get_default_mode()
+ if default_mode_value is None:
+ default_mode_from_conf =
spark.conf.get("spark.sql.pyspark.pandas.defaultMode") == "true"
Review Comment:
Shall we add the default value for `get`? Otherwise it will fail with old
versions of Spark where the new config is not available.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala:
##########
@@ -278,4 +278,12 @@ object StaticSQLConf {
.version("3.1.0")
.stringConf
.createWithDefault("")
+
+ val DEFAULT_MODE = buildStaticConf("spark.sql.pyspark.pandas.defaultMode")
+ .doc("When set to true, set_option and reset_option will all be ignored,
and only default" +
Review Comment:
nit: add a space at the end of the string?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
