michaelzhan-db commented on code in PR #42548:
URL: https://github.com/apache/spark/pull/42548#discussion_r1300667372
##########
python/pyspark/testing/connectutils.py:
##########
@@ -167,6 +167,7 @@ def conf(cls):
Override this in subclasses to supply a more specific conf
"""
conf = SparkConf(loadDefaults=False)
+ conf.set("spark.sql.legacy.setCommandRejectsSparkCoreConfs", "false")
Review Comment:
ConfigEntry's are added when they are created. [maxStreamDuration
configuration option
creation.](https://github.com/apache/spark/blob/b9c4fa4bc3129a352c443be04ef88f3ea71ae99d/connector/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala#L112).
[ConfigEntry constructor registers the config entry into the global config
entry
registry.](https://github.com/apache/spark/blob/b9c4fa4bc3129a352c443be04ef88f3ea71ae99d/core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala#L87)
[RuntimeConf configuration
checking](https://github.com/apache/spark/blob/b9c4fa4bc3129a352c443be04ef88f3ea71ae99d/sql/core/src/main/scala/org/apache/spark/sql/RuntimeConfig.scala#L160)
checks [spark.sql.legacy.setCommandRejectsSparkCoreConfs which defaults to
true](https://github.com/apache/spark/blob/b9c4fa4bc3129a352c443be04ef88f3ea71ae99d/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L3672).
Note that since `spark.sql.legacy.setCommandRejectsSparkCoreConfs` is not a
SQLConf, it is not found in the SQLConf registry so `requireNonStatic` throws.
I'm using `spark.sql.legacy.setCommandRejectsSparkCoreConfs` to set non-sql
configurations to get the test to run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
