[
https://issues.apache.org/jira/browse/SPARK-61047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61047:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Warn the warnings of the server in spark.conf.set and unset, and take the
> arguments of get and set, in the Python client as PySpark does
> ----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61047
> URL: https://issues.apache.org/jira/browse/SPARK-61047
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> In PySpark Connect, {{spark.conf.set}} and {{spark.conf.unset}}, and the
> options of {{SparkSession.builder.config}}, warn each warning of the server
> as a {{UserWarning}}, such as "The SQL config
> 'spark.sql.execution.arrow.enabled' is NOT supported in Spark Connect".
> {{spark.conf.get}} raises {{NOT_EXPECTED_TYPE}} of {{key}} for a key that is
> not a str, and of {{default}} for a default that is neither None nor a str.
> {{spark.conf.set}} takes a str, a bool or an int: it sends None as no value,
> which the server rejects, and raises a {{TypeError}} for a value of another
> type, such as a float.
> The Rust-backed Python client drops the warnings of the server, raises
> {{TypeError}}s for the arguments of {{get}}, and sets the {{str()}} of any
> value: {{spark.conf.set(key, None)}} sets "null", and {{spark.conf.set(key,
> 1.5)}} sets "1.5". Take them as PySpark does.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
