Hyukjin Kwon created SPARK-61047:
------------------------------------
Summary: Warn the warnings of the server in spark.conf.set and
unset, and take the arguments of get and set, in the Python client as PySpark
does
Key: SPARK-61047
URL: https://issues.apache.org/jira/browse/SPARK-61047
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
In PySpark Connect, {{spark.conf.set}} and {{spark.conf.unset}}, and the
options of {{SparkSession.builder.config}}, warn each warning of the server as
a {{UserWarning}}, such as "The SQL config 'spark.sql.execution.arrow.enabled'
is NOT supported in Spark Connect". {{spark.conf.get}} raises
{{NOT_EXPECTED_TYPE}} of {{key}} for a key that is not a str, and of
{{default}} for a default that is neither None nor a str. {{spark.conf.set}}
takes a str, a bool or an int: it sends None as no value, which the server
rejects, and raises a {{TypeError}} for a value of another type, such as a
float.
The Rust-backed Python client drops the warnings of the server, raises
{{TypeError}}s for the arguments of {{get}}, and sets the {{str()}} of any
value: {{spark.conf.set(key, None)}} sets "null", and {{spark.conf.set(key,
1.5)}} sets "1.5". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
