This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 64b25a76 [SPARK-61169][CONNECT][PYTHON] Add get_configs,
get_config_dict and get_config_with_defaults to SparkConnectClient, of which
pandas-on-Spark reads its configurations, in the Python client as PySpark has
them
64b25a76 is described below
commit 64b25a76a08026dd8c48ed5dc53b5986b302a0a4
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 17:25:53 2026 +0900
[SPARK-61169][CONNECT][PYTHON] Add get_configs, get_config_dict and
get_config_with_defaults to SparkConnectClient, of which pandas-on-Spark reads
its configurations, in the Python client as PySpark has them
### What changes were proposed in this pull request?
[SPARK-61169](https://issues.apache.org/jira/browse/SPARK-61169): PySpark
Connect's `SparkConnectClient` (`spark.client`) has three methods that read
configurations, each in one `Config` request:
- `get_configs(*keys)`, of `Get`
- `get_config_dict(*keys)`, of `Get`
- `get_config_with_defaults(*pairs)`, of `GetWithDefault` for the pairs of
a key and its default
pandas-on-Spark's `is_ansi_mode_enabled` calls `get_config_with_defaults`.
It reads its ANSI mode so to show a Series or a DataFrame, and in `to_numeric`
and other functions.
The client's `SparkConnectClient` now has the three methods, also one
request each:
- **Rust core**: `RuntimeConfig::get_with_defaults` reads the values of
keys of defaults in one request, as `RuntimeConfig::get_many` does of keys.
- **Python client**: the methods read the configurations through private
methods of the client's `RuntimeConfig` (`_get_many`, `_get_with_defaults`).
### Why are the changes needed?
The client's `SparkConnectClient` had none of them:
```
AttributeError: 'SparkConnectClient' object has no attribute
'get_config_with_defaults'
```
Three tests of Apache Spark's pandas-on-Spark failed on this client:
`test_to_numeric` of `test_parity_namespace.py`, `test_repr_series` of
`test_parity_repr.py`, and `test_creation_index_same_anchor` of
`frame/test_parity_constructor.py`.
### Does this PR introduce _any_ user-facing change?
Yes. `spark.client` has the configuration methods of PySpark 4.3.0, and
pandas-on-Spark shows its Series and DataFrames under them. For the Rust API,
`RuntimeConfig::get_with_defaults` is new.
### How was this patch tested?
- A new live test checks:
- the values of the three methods, of a key that is set, a key that is
not set with a default and one without
- the `SparkNoSuchElementException` `SQL_CONF_NOT_FOUND` of `get_configs`
of a key that is not set
- the repr of a pandas-on-Spark Series
It passes with official PySpark 4.3.0, and fails without this change.
- The values and the error against a local server are those of official
PySpark 4.3.0.
- Apache Spark's `test_parity_namespace.py`, `test_parity_repr.py` and
`frame/test_parity_constructor.py`, of pandas-on-Spark, pass on this client
against a local server: 17, 7 and 11 tests.
- The lib tests of both crates (599 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2224 passed, 8 skipped), and the
live drop-in and Arrow suites (208 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #362 from HyukjinKwon/SPARK-61169-client-configs.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/conf.rs | 28 +++++++++++++++++++++++++++
crates/spark-connect/src/conf.rs | 7 +++++++
python/pyspark/sql/connect/client/__init__.py | 20 ++++++++++++++++++-
python/tests/test_dropin_e2e.py | 26 +++++++++++++++++++++++++
4 files changed, 80 insertions(+), 1 deletion(-)
diff --git a/crates/pyspark-rs/src/conf.rs b/crates/pyspark-rs/src/conf.rs
index dfebb2ad..156a9216 100644
--- a/crates/pyspark-rs/src/conf.rs
+++ b/crates/pyspark-rs/src/conf.rs
@@ -151,6 +151,34 @@ impl PyRuntimeConf {
}
/// Whether the given configuration key is modifiable at runtime.
+ /// The values of the configurations `keys`, in one request, as PySpark
Connect's
+ /// `SparkConnectClient.get_configs`, which `SparkConnectClient` of the
connect import path
+ /// gives.
+ fn _get_many(&self, py: Python<'_>, keys: Vec<String>) ->
PyResult<Vec<Option<String>>> {
+ py.detach(|| {
+ let keys: Vec<&str> = keys.iter().map(String::as_str).collect();
+ self.conf.get_many(&keys).map_err(Box::new)
+ })
+ .to_pyerr()
+ }
+
+ /// The values of the configurations of `pairs` of a key and its default,
in one request, as
+ /// PySpark Connect's `SparkConnectClient.get_config_with_defaults`.
+ fn _get_with_defaults(
+ &self,
+ py: Python<'_>,
+ pairs: Vec<(String, Option<String>)>,
+ ) -> PyResult<Vec<Option<String>>> {
+ py.detach(|| {
+ let pairs: Vec<(&str, Option<&str>)> = pairs
+ .iter()
+ .map(|(key, default)| (key.as_str(), default.as_deref()))
+ .collect();
+ self.conf.get_with_defaults(&pairs).map_err(Box::new)
+ })
+ .to_pyerr()
+ }
+
#[pyo3(name = "isModifiable")]
fn is_modifiable(&self, py: Python<'_>, key: &str) -> PyResult<bool> {
py.detach(|| self.conf.is_modifiable(key)).to_pyerr()
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index c742c935..ff2114bc 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -93,6 +93,13 @@ impl RuntimeConf {
Ok(results.into_iter().next().flatten())
}
+ /// Get the values of configurations, each of a default, in one request,
of the server's
+ /// `GetWithDefault` operation: the default of a key that is not set.
Mirrors PySpark
+ /// Connect's `SparkConnectClient.get_config_with_defaults`.
+ pub fn get_with_defaults(&self, pairs: &[(&str, Option<&str>)]) ->
Result<Vec<Option<String>>> {
+ block_on(self.client.get_config_with_defaults(pairs))
+ }
+
/// Get all configuration values as a HashMap.
///
/// # Example
diff --git a/python/pyspark/sql/connect/client/__init__.py
b/python/pyspark/sql/connect/client/__init__.py
index 6b9152c3..0774ad42 100644
--- a/python/pyspark/sql/connect/client/__init__.py
+++ b/python/pyspark/sql/connect/client/__init__.py
@@ -22,7 +22,7 @@ __all__ = ["SparkConnectClient"]
import weakref
-from typing import Any, Dict, List
+from typing import Any, Dict, List, Mapping, Optional, Tuple
from pyspark.sql.metrics import ObservedMetrics
@@ -145,6 +145,24 @@ class SparkConnectClient:
whether its channel is closed."""
return self._session.is_stopped
+ def get_configs(self, *keys: str) -> Tuple[Optional[str], ...]:
+ """The values of the configurations ``keys``, in one request, as
SparkConnectClient's
+ ``get_configs``."""
+ return tuple(self._session.conf._get_many(list(keys)))
+
+ def get_config_dict(self, *keys: str) -> Mapping[str, Optional[str]]:
+ """The values of the configurations ``keys`` by their keys, in one
request, as
+ SparkConnectClient's ``get_config_dict``."""
+ return dict(zip(keys, self.get_configs(*keys)))
+
+ def get_config_with_defaults(
+ self, *pairs: Tuple[str, Optional[str]]
+ ) -> Tuple[Optional[str], ...]:
+ """The values of the configurations of ``pairs`` of a key and its
default, the default of
+ a key that is not set, in one request, as SparkConnectClient's
+ ``get_config_with_defaults``, of which pandas-on-Spark reads its ANSI
mode."""
+ return tuple(self._session.conf._get_with_defaults(list(pairs)))
+
def execute_command(self, command: Any, observations: Any = None) -> Any:
"""Execute ``command``, a ``spark.connect.Command``, as
SparkConnectClient's
``execute_command``: no data, the properties of the results of the
responses, of their
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index bc91ba74..02e3c584 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -165,6 +165,32 @@ def test_conf_get_with_a_none_default(session):
session.conf.get(key)
+def test_client_reads_configurations_as_pyspark(session):
+ # As PySpark Connect's SparkConnectClient, of which pandas-on-Spark reads
its ANSI mode: the
+ # values of keys, by keys, and of keys of defaults, which a key that is
not set takes.
+ from pyspark.errors import SparkNoSuchElementException
+
+ client = session.client
+ key, unset = "spark.connect.rust.test.client.set",
"spark.connect.rust.test.client.unset"
+ session.conf.set(key, "value")
+ try:
+ assert client.get_config_with_defaults(
+ (key, None), (unset, "default"), (unset + ".none", None)
+ ) == ("value", "default", None)
+ assert client.get_configs(key) == ("value",)
+ assert client.get_config_dict(key) == {key: "value"}
+ finally:
+ session.conf.unset(key)
+ with pytest.raises(SparkNoSuchElementException) as error:
+ client.get_configs(unset)
+ assert error.value.getCondition() == "SQL_CONF_NOT_FOUND"
+ # pandas-on-Spark reads its ANSI mode so to show a Series.
+ pd = pytest.importorskip("pandas")
+ import pyspark.pandas as ps
+
+ assert repr(ps.Series([1, 2, 3], name="x")) == repr(pd.Series([1, 2, 3],
name="x"))
+
+
def
test_numpy_scalar_literals_have_the_types_of_their_dtypes_as_pyspark(session):
np = pytest.importorskip("numpy")
from pyspark.sql import functions as F
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]