[
https://issues.apache.org/jira/browse/SPARK-61169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-61169:
---------------------------------
Fix Version/s: connect-rust-4.3.0
> Add get_configs, get_config_dict and get_config_with_defaults to
> SparkConnectClient, of which pandas-on-Spark reads its configurations, in the
> Python client as PySpark has them
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61169
> URL: https://issues.apache.org/jira/browse/SPARK-61169
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect's SparkConnectClient (spark.client) has get_configs(*keys),
> get_config_dict(*keys) and get_config_with_defaults(*pairs), each one Config
> request: of Get, and of GetWithDefault for the pairs of a key and its
> default. pandas-on-Spark's is_ansi_mode_enabled calls
> get_config_with_defaults, which it reads to show a Series or a DataFrame, and
> in to_numeric and other functions.
> The Python client's SparkConnectClient has none of them:
> AttributeError: 'SparkConnectClient' object has no attribute
> 'get_config_with_defaults'
> Apache Spark's test_to_numeric of test_parity_namespace.py, test_repr_series
> of test_parity_repr.py and test_creation_index_same_anchor of
> frame/test_parity_constructor.py, of pandas-on-Spark, fail on this client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
