This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 64b25a76 [SPARK-61169][CONNECT][PYTHON] Add get_configs, 
get_config_dict and get_config_with_defaults to SparkConnectClient, of which 
pandas-on-Spark reads its configurations, in the Python client as PySpark has 
them
64b25a76 is described below

commit 64b25a76a08026dd8c48ed5dc53b5986b302a0a4
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 17:25:53 2026 +0900

    [SPARK-61169][CONNECT][PYTHON] Add get_configs, get_config_dict and 
get_config_with_defaults to SparkConnectClient, of which pandas-on-Spark reads 
its configurations, in the Python client as PySpark has them
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61169](https://issues.apache.org/jira/browse/SPARK-61169): PySpark 
Connect's `SparkConnectClient` (`spark.client`) has three methods that read 
configurations, each in one `Config` request:
    - `get_configs(*keys)`, of `Get`
    - `get_config_dict(*keys)`, of `Get`
    - `get_config_with_defaults(*pairs)`, of `GetWithDefault` for the pairs of 
a key and its default
    
    pandas-on-Spark's `is_ansi_mode_enabled` calls `get_config_with_defaults`. 
It reads its ANSI mode so to show a Series or a DataFrame, and in `to_numeric` 
and other functions.
    
    The client's `SparkConnectClient` now has the three methods, also one 
request each:
    - **Rust core**: `RuntimeConfig::get_with_defaults` reads the values of 
keys of defaults in one request, as `RuntimeConfig::get_many` does of keys.
    - **Python client**: the methods read the configurations through private 
methods of the client's `RuntimeConfig` (`_get_many`, `_get_with_defaults`).
    
    ### Why are the changes needed?
    
    The client's `SparkConnectClient` had none of them:
    
    ```
    AttributeError: 'SparkConnectClient' object has no attribute 
'get_config_with_defaults'
    ```
    
    Three tests of Apache Spark's pandas-on-Spark failed on this client: 
`test_to_numeric` of `test_parity_namespace.py`, `test_repr_series` of 
`test_parity_repr.py`, and `test_creation_index_same_anchor` of 
`frame/test_parity_constructor.py`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `spark.client` has the configuration methods of PySpark 4.3.0, and 
pandas-on-Spark shows its Series and DataFrames under them. For the Rust API, 
`RuntimeConfig::get_with_defaults` is new.
    
    ### How was this patch tested?
    
    - A new live test checks:
      - the values of the three methods, of a key that is set, a key that is 
not set with a default and one without
      - the `SparkNoSuchElementException` `SQL_CONF_NOT_FOUND` of `get_configs` 
of a key that is not set
      - the repr of a pandas-on-Spark Series
    
      It passes with official PySpark 4.3.0, and fails without this change.
    - The values and the error against a local server are those of official 
PySpark 4.3.0.
    - Apache Spark's `test_parity_namespace.py`, `test_parity_repr.py` and 
`frame/test_parity_constructor.py`, of pandas-on-Spark, pass on this client 
against a local server: 17, 7 and 11 tests.
    - The lib tests of both crates (599 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2224 passed, 8 skipped), and the 
live drop-in and Arrow suites (208 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #362 from HyukjinKwon/SPARK-61169-client-configs.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/conf.rs                 | 28 +++++++++++++++++++++++++++
 crates/spark-connect/src/conf.rs              |  7 +++++++
 python/pyspark/sql/connect/client/__init__.py | 20 ++++++++++++++++++-
 python/tests/test_dropin_e2e.py               | 26 +++++++++++++++++++++++++
 4 files changed, 80 insertions(+), 1 deletion(-)

diff --git a/crates/pyspark-rs/src/conf.rs b/crates/pyspark-rs/src/conf.rs
index dfebb2ad..156a9216 100644
--- a/crates/pyspark-rs/src/conf.rs
+++ b/crates/pyspark-rs/src/conf.rs
@@ -151,6 +151,34 @@ impl PyRuntimeConf {
     }
 
     /// Whether the given configuration key is modifiable at runtime.
+    /// The values of the configurations `keys`, in one request, as PySpark 
Connect's
+    /// `SparkConnectClient.get_configs`, which `SparkConnectClient` of the 
connect import path
+    /// gives.
+    fn _get_many(&self, py: Python<'_>, keys: Vec<String>) -> 
PyResult<Vec<Option<String>>> {
+        py.detach(|| {
+            let keys: Vec<&str> = keys.iter().map(String::as_str).collect();
+            self.conf.get_many(&keys).map_err(Box::new)
+        })
+        .to_pyerr()
+    }
+
+    /// The values of the configurations of `pairs` of a key and its default, 
in one request, as
+    /// PySpark Connect's `SparkConnectClient.get_config_with_defaults`.
+    fn _get_with_defaults(
+        &self,
+        py: Python<'_>,
+        pairs: Vec<(String, Option<String>)>,
+    ) -> PyResult<Vec<Option<String>>> {
+        py.detach(|| {
+            let pairs: Vec<(&str, Option<&str>)> = pairs
+                .iter()
+                .map(|(key, default)| (key.as_str(), default.as_deref()))
+                .collect();
+            self.conf.get_with_defaults(&pairs).map_err(Box::new)
+        })
+        .to_pyerr()
+    }
+
     #[pyo3(name = "isModifiable")]
     fn is_modifiable(&self, py: Python<'_>, key: &str) -> PyResult<bool> {
         py.detach(|| self.conf.is_modifiable(key)).to_pyerr()
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index c742c935..ff2114bc 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -93,6 +93,13 @@ impl RuntimeConf {
         Ok(results.into_iter().next().flatten())
     }
 
+    /// Get the values of configurations, each of a default, in one request, 
of the server's
+    /// `GetWithDefault` operation: the default of a key that is not set. 
Mirrors PySpark
+    /// Connect's `SparkConnectClient.get_config_with_defaults`.
+    pub fn get_with_defaults(&self, pairs: &[(&str, Option<&str>)]) -> 
Result<Vec<Option<String>>> {
+        block_on(self.client.get_config_with_defaults(pairs))
+    }
+
     /// Get all configuration values as a HashMap.
     ///
     /// # Example
diff --git a/python/pyspark/sql/connect/client/__init__.py 
b/python/pyspark/sql/connect/client/__init__.py
index 6b9152c3..0774ad42 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -22,7 +22,7 @@ __all__ = ["SparkConnectClient"]
 
 
 import weakref
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Mapping, Optional, Tuple
 
 from pyspark.sql.metrics import ObservedMetrics
 
@@ -145,6 +145,24 @@ class SparkConnectClient:
         whether its channel is closed."""
         return self._session.is_stopped
 
+    def get_configs(self, *keys: str) -> Tuple[Optional[str], ...]:
+        """The values of the configurations ``keys``, in one request, as 
SparkConnectClient's
+        ``get_configs``."""
+        return tuple(self._session.conf._get_many(list(keys)))
+
+    def get_config_dict(self, *keys: str) -> Mapping[str, Optional[str]]:
+        """The values of the configurations ``keys`` by their keys, in one 
request, as
+        SparkConnectClient's ``get_config_dict``."""
+        return dict(zip(keys, self.get_configs(*keys)))
+
+    def get_config_with_defaults(
+        self, *pairs: Tuple[str, Optional[str]]
+    ) -> Tuple[Optional[str], ...]:
+        """The values of the configurations of ``pairs`` of a key and its 
default, the default of
+        a key that is not set, in one request, as SparkConnectClient's
+        ``get_config_with_defaults``, of which pandas-on-Spark reads its ANSI 
mode."""
+        return tuple(self._session.conf._get_with_defaults(list(pairs)))
+
     def execute_command(self, command: Any, observations: Any = None) -> Any:
         """Execute ``command``, a ``spark.connect.Command``, as 
SparkConnectClient's
         ``execute_command``: no data, the properties of the results of the 
responses, of their
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index bc91ba74..02e3c584 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -165,6 +165,32 @@ def test_conf_get_with_a_none_default(session):
         session.conf.get(key)
 
 
+def test_client_reads_configurations_as_pyspark(session):
+    # As PySpark Connect's SparkConnectClient, of which pandas-on-Spark reads 
its ANSI mode: the
+    # values of keys, by keys, and of keys of defaults, which a key that is 
not set takes.
+    from pyspark.errors import SparkNoSuchElementException
+
+    client = session.client
+    key, unset = "spark.connect.rust.test.client.set", 
"spark.connect.rust.test.client.unset"
+    session.conf.set(key, "value")
+    try:
+        assert client.get_config_with_defaults(
+            (key, None), (unset, "default"), (unset + ".none", None)
+        ) == ("value", "default", None)
+        assert client.get_configs(key) == ("value",)
+        assert client.get_config_dict(key) == {key: "value"}
+    finally:
+        session.conf.unset(key)
+    with pytest.raises(SparkNoSuchElementException) as error:
+        client.get_configs(unset)
+    assert error.value.getCondition() == "SQL_CONF_NOT_FOUND"
+    # pandas-on-Spark reads its ANSI mode so to show a Series.
+    pd = pytest.importorskip("pandas")
+    import pyspark.pandas as ps
+
+    assert repr(ps.Series([1, 2, 3], name="x")) == repr(pd.Series([1, 2, 3], 
name="x"))
+
+
 def 
test_numpy_scalar_literals_have_the_types_of_their_dtypes_as_pyspark(session):
     np = pytest.importorskip("numpy")
     from pyspark.sql import functions as F


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to