This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 0c39208 [SPARK-61047][CONNECT][PYTHON] Warn the warnings of the
server in spark.conf.set and unset, and take the arguments of get and set, in
the Python client as PySpark does
0c39208 is described below
commit 0c392087a29793917f8f93deb6961f68c8cfaa7b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sat Nov 7 11:51:10 2026 +0900
[SPARK-61047][CONNECT][PYTHON] Warn the warnings of the server in
spark.conf.set and unset, and take the arguments of get and set, in the Python
client as PySpark does
### What changes were proposed in this pull request?
[SPARK-61047](https://issues.apache.org/jira/browse/SPARK-61047):
`spark.conf` now warns the warnings of the server, and takes its arguments, as
PySpark Connect's `RuntimeConf` does.
- `spark.conf.set`, `spark.conf.unset` and the options of
`SparkSession.builder.config` warn each warning of the server as a
`UserWarning`, such as "The SQL config 'spark.sql.execution.arrow.enabled' is
NOT supported in Spark Connect".
- `spark.conf.get` raises `NOT_EXPECTED_TYPE` of `key` for a key that is
not a str. It raises it of `default` for a default that is neither None nor a
str.
- `spark.conf.set` sends:
- a bool as "true" or "false"
- an int as its `str()`
- a str, or UTF-8 bytes, as it is
- None as no value, which the server rejects
It raises the `TypeError` of protobuf for a value of another type, such
as a float.
The core `RuntimeConf::set`, `set_all` and `unset`, and the
`SparkConnectClient` methods they call, now return the warnings of the server.
The new `RuntimeConf::set_pairs` takes values that may be None.
The builder warns its warnings once. PySpark warns them twice when it
creates a session, as it sets its options twice; under the default warning
filter, both show one.
### Why are the changes needed?
The client:
- dropped the warnings of the server, so a deprecated or unsupported
configuration was set silently
- raised `TypeError`s for the arguments of `get`
- set the `str()` of any value: `spark.conf.set(key, None)` set "null", and
`spark.conf.set(key, 1.5)` set "1.5"
### Does this PR introduce _any_ user-facing change?
Yes. `spark.conf` warns, and takes values, as in PySpark 4.3.0.
### How was this patch tested?
- A new live test checks the warnings of `set` and `unset` of an
unsupported configuration, the rejection of None by the server, the `TypeError`
of a float, and the errors of `get`. It passes with official PySpark 4.3.0, and
fails without this change.
- A new offline test checks the warnings of `set`, `unset` and the builder,
the values that `set` sends, and the errors of `set` and `get`, against a mock
Connect server. The mock server now answers `Unset`, returns the warnings of a
test, and records a value that is not set.
- On a local server, 42 cases of `spark.conf` give the results, errors and
warnings of official PySpark 4.3.0. The exceptions are the warnings of the
builder, as above, and the messages of the `TypeError`s of 3 keys that are not
a str.
- The lib tests of both crates (571 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the
live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #247 from HyukjinKwon/SPARK-61047-conf.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/conf.rs | 119 +++++++++++++++++++++++++++-----
crates/pyspark-rs/src/session.rs | 6 +-
crates/spark-connect-core/src/client.rs | 72 +++++++++----------
crates/spark-connect/src/conf.rs | 34 ++++++---
python/tests/_mock_connect_server.py | 23 +++++-
python/tests/test_dropin_e2e.py | 25 +++++++
python/tests/test_dropin_offline.py | 56 +++++++++++++++
7 files changed, 267 insertions(+), 68 deletions(-)
diff --git a/crates/pyspark-rs/src/conf.rs b/crates/pyspark-rs/src/conf.rs
index f3eb8dd..dfebb2a 100644
--- a/crates/pyspark-rs/src/conf.rs
+++ b/crates/pyspark-rs/src/conf.rs
@@ -21,19 +21,71 @@ impl PyRuntimeConf {
/// from None: `get(key, None)` returns None for an unset key, while
`get(key)` raises.
pub enum ConfDefault {
Omitted,
- Given(Option<String>),
+ Given(Py<PyAny>),
}
impl<'a, 'py> FromPyObject<'a, 'py> for ConfDefault {
type Error = PyErr;
fn extract(obj: Borrowed<'a, 'py, PyAny>) -> PyResult<Self> {
- if obj.is_none() {
- Ok(ConfDefault::Given(None))
- } else {
- Ok(ConfDefault::Given(Some(obj.extract()?)))
- }
+ Ok(ConfDefault::Given(obj.to_owned().unbind()))
+ }
+}
+
+/// `obj`, a str, else PySpark's `NOT_EXPECTED_TYPE` of the argument `name`, as
+/// `RuntimeConf._checkType`.
+fn checked_str(obj: &Bound<'_, PyAny>, name: &str) -> PyResult<String> {
+ match obj.cast::<pyo3::types::PyString>() {
+ Ok(text) => Ok(text.to_cow()?.into_owned()),
+ Err(_) => Err(crate::stat_arguments::not_expected_type(obj, name,
"str")),
+ }
+}
+
+/// The value of `RuntimeConf.set`, as PySpark sends it in a `KeyValue`: a
bool as "true" or
+/// "false", an int as its `str()`, a str, or UTF-8 bytes, as it is, and None
as no value,
+/// which the server rejects. Another value is the `TypeError` of protobuf, as
in PySpark.
+fn conf_value(value: &Bound<'_, PyAny>) -> PyResult<Option<String>> {
+ use pyo3::types::{PyBool, PyBytes, PyInt, PyString};
+ if value.is_none() {
+ return Ok(None);
+ }
+ if let Ok(flag) = value.cast::<PyBool>() {
+ return Ok(Some(
+ if flag.is_true() { "true" } else { "false" }.to_string(),
+ ));
+ }
+ if value.is_instance_of::<PyInt>() {
+ return Ok(Some(value.str()?.to_cow()?.into_owned()));
+ }
+ if let Ok(text) = value.cast::<PyString>() {
+ return Ok(Some(text.to_cow()?.into_owned()));
+ }
+ let shown = value.repr()?;
+ if let Ok(bytes) = value.cast::<PyBytes>() {
+ return String::from_utf8(bytes.as_bytes().to_vec())
+ .map(Some)
+ .map_err(|_| {
+ pyo3::exceptions::PyValueError::new_err(format!(
+ "{shown} has type bytes, but isn't valid UTF-8 encoding.
Non-UTF-8 strings \
+ must be converted to unicode objects before being added."
+ ))
+ });
+ }
+ Err(pyo3::exceptions::PyTypeError::new_err(format!(
+ "Cannot set spark.connect.KeyValue.value to {shown}: {shown} has type
{}, but expected \
+ one of: (<class 'bytes'>, <class 'str'>) for field KeyValue.value",
+ value.get_type().repr()?
+ )))
+}
+
+/// Warn each warning of the server, such as of a deprecated or unsupported
configuration, as
+/// PySpark's `RuntimeConf` does with `warnings.warn`.
+pub(crate) fn warn_all(py: Python<'_>, warnings: &[String]) -> PyResult<()> {
+ let warn = py.import("warnings")?.getattr("warn")?;
+ for warning in warnings {
+ warn.call1((warning,))?;
}
+ Ok(())
}
// Every method round-trips to the server
(`block_on(client.set_config/get_configs)`
@@ -41,30 +93,61 @@ impl<'a, 'py> FromPyObject<'a, 'py> for ConfDefault {
// Python thread stays blocked for the whole call.
#[pymethods]
impl PyRuntimeConf {
- /// Set a configuration value. A bool lowercases to "true"/"false"
(reference
- /// `to_str`); other values use their `str()`. (`None` would be unusual
here;
- /// coerce_option_value maps it to no value, so fall back to "null".)
+ /// Set a configuration value, as PySpark's: a bool as "true" or "false",
an int as its
+ /// `str()`, a str as it is, and None as no value, which the server
rejects. Warns each
+ /// warning of the server.
fn set(&self, py: Python<'_>, key: &str, value: &Bound<'_, PyAny>) ->
PyResult<()> {
- let v = crate::coerce_option_value(value)?.unwrap_or_else(||
"null".to_string());
- py.detach(|| self.conf.set(key, &v)).to_pyerr()
+ let value = conf_value(value)?;
+ let warnings = py
+ .detach(|| {
+ self.conf
+ .set_pairs(&[(key, value.as_deref())], None)
+ .map_err(Box::new)
+ })
+ .to_pyerr()?;
+ warn_all(py, &warnings)
}
/// Get a configuration value. With a `default`, which may be None, an
unknown key returns
/// the default (server `GetWithDefault`) rather than raising; without
one, an unknown key
- /// raises, mirroring `RuntimeConfig.get`.
+ /// raises, mirroring `RuntimeConfig.get`, which raises
`NOT_EXPECTED_TYPE` for a key, and
+ /// a default other than None, that is not a str.
#[pyo3(signature = (key, default=ConfDefault::Omitted))]
- fn get(&self, py: Python<'_>, key: &str, default: ConfDefault) ->
PyResult<Option<String>> {
+ fn get(
+ &self,
+ py: Python<'_>,
+ key: &Bound<'_, PyAny>,
+ default: ConfDefault,
+ ) -> PyResult<Option<String>> {
+ let key = checked_str(key, "key")?;
match default {
- ConfDefault::Given(d) => py
- .detach(|| self.conf.get_with_default(key, d.as_deref()))
+ ConfDefault::Given(default) => {
+ let default = default.bind(py);
+ let default = if default.is_none() {
+ None
+ } else {
+ Some(checked_str(default, "default")?)
+ };
+ py.detach(|| {
+ self.conf
+ .get_with_default(&key, default.as_deref())
+ .map_err(Box::new)
+ })
+ .to_pyerr()
+ }
+ ConfDefault::Omitted => py
+ .detach(|| self.conf.get(&key).map_err(Box::new))
.to_pyerr(),
- ConfDefault::Omitted => py.detach(||
self.conf.get(key)).to_pyerr(),
}
}
- /// Reset a configuration value to its default.
+ /// Reset a configuration value to its default. Warns each warning of the
server, as
+ /// PySpark's.
fn unset(&self, py: Python<'_>, key: &str) -> PyResult<()> {
- py.detach(|| self.conf.unset(key)).to_pyerr()
+ let warnings = py
+ .detach(|| self.conf.unset(key).map_err(Box::new))
+ .to_pyerr()?;
+ warn_all(py, &warnings)
}
/// Whether the given configuration key is modifiable at runtime.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 05f8960..d039aee 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -261,7 +261,11 @@ impl PySparkSessionBuilder {
return Ok(());
}
let conf = session.conf();
- py.detach(|| conf.set_all(&options, true)).to_pyerr()
+ let warnings = py
+ .detach(|| conf.set_all(&options, true).map_err(Box::new))
+ .to_pyerr()?;
+ // As PySpark's `_set_all`, which warns each warning of the server.
+ crate::conf::warn_all(py, &warnings)
}
}
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 7d68f87..9e26346 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -753,46 +753,49 @@ impl SparkConnectClient {
Ok(keys.iter().map(|k| config_dict.get(*k).cloned()).collect())
}
- /// Set configuration values for the session.
+ /// Set configuration values for the session, returning the warnings of
the server, such
+ /// as of a deprecated or unsupported configuration.
///
/// Mirrors
`pyspark.sql.connect.client.core.SparkConnectClient.set_config`.
- pub async fn set_config(&self, key: &str, value: &str) -> Result<()> {
- let mut request = ConfigRequest::default();
- request.session_id = self.session_id.clone();
- request.user_context = Some(UserContext {
- user_id: self.user_id.clone().unwrap_or_default(),
- ..Default::default()
- });
-
- let mut set = spark_connect_proto::config_request::Set::default();
- set.pairs.push(spark_connect_proto::KeyValue {
- key: key.to_string(),
- value: Some(value.to_string()),
- });
-
- let mut operation =
spark_connect_proto::config_request::Operation::default();
- operation.op_type =
Some(spark_connect_proto::config_request::operation::OpType::Set(
- set,
- ));
- request.operation = Some(operation);
-
- let _ = self.config(request).await?;
- Ok(())
+ pub async fn set_config(&self, key: &str, value: &str) ->
Result<Vec<String>> {
+ self.set_config_values(&[(key, Some(value))], None).await
}
/// Set configuration values in one request, ignoring the failures to set
them when
- /// `silent`. Mirrors `pyspark.sql.connect.conf.RuntimeConf._set_all`,
with which the
- /// session builder applies its options.
- pub async fn set_configs(&self, pairs: &[(String, String)], silent: bool)
-> Result<()> {
+ /// `silent`, returning the warnings of the server. Mirrors
+ /// `pyspark.sql.connect.conf.RuntimeConf._set_all`, with which the
session builder applies
+ /// its options.
+ pub async fn set_configs(
+ &self,
+ pairs: &[(String, String)],
+ silent: bool,
+ ) -> Result<Vec<String>> {
+ let pairs: Vec<(&str, Option<&str>)> = pairs
+ .iter()
+ .map(|(key, value)| (key.as_str(), Some(value.as_str())))
+ .collect();
+ self.set_config_values(&pairs, Some(silent)).await
+ }
+
+ /// Set configuration values in one request, as PySpark's
`RuntimeConf.set` and `_set_all`
+ /// send them: a pair without a value is sent without one, as PySpark
sends None, which the
+ /// server rejects, and with `silent` the failures to set them, such as of
static
+ /// configurations, are ignored. Returns the warnings of the server, such
as of a
+ /// deprecated or unsupported configuration, which PySpark warns.
+ pub async fn set_config_values(
+ &self,
+ pairs: &[(&str, Option<&str>)],
+ silent: Option<bool>,
+ ) -> Result<Vec<String>> {
let set = spark_connect_proto::config_request::Set {
pairs: pairs
.iter()
.map(|(key, value)| spark_connect_proto::KeyValue {
- key: key.clone(),
- value: Some(value.clone()),
+ key: key.to_string(),
+ value: value.map(str::to_string),
})
.collect(),
- silent: Some(silent),
+ silent,
};
let request = ConfigRequest {
session_id: self.session_id.clone(),
@@ -807,8 +810,7 @@ impl SparkConnectClient {
}),
..Default::default()
};
- let _ = self.config(request).await?;
- Ok(())
+ Ok(self.config(request).await?.warnings)
}
/// Get configuration values with default fallbacks.
@@ -857,10 +859,11 @@ impl SparkConnectClient {
.collect())
}
- /// Unset a configuration value.
+ /// Unset a configuration value, returning the warnings of the server,
such as of a
+ /// deprecated or unsupported configuration.
///
/// Mirrors
`pyspark.sql.connect.client.core.SparkConnectClient.unset_config`.
- pub async fn unset_config(&self, key: &str) -> Result<()> {
+ pub async fn unset_config(&self, key: &str) -> Result<Vec<String>> {
let mut request = ConfigRequest::default();
request.session_id = self.session_id.clone();
request.user_context = Some(UserContext {
@@ -876,8 +879,7 @@ impl SparkConnectClient {
Some(spark_connect_proto::config_request::operation::OpType::Unset(unset));
request.operation = Some(operation);
- let _ = self.config(request).await?;
- Ok(())
+ Ok(self.config(request).await?.warnings)
}
/// Get all configuration values from the session.
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index 10d38c1..c742c93 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -23,32 +23,43 @@ impl RuntimeConf {
RuntimeConf { client }
}
- /// Set a configuration key-value pair.
- ///
- /// Converts boolean and integer values to strings as required by the
server.
+ /// Set a configuration key-value pair, returning the warnings of the
server, such as of a
+ /// deprecated or unsupported configuration, which PySpark's
`RuntimeConf.set` warns.
///
/// # Arguments
///
/// * `key` - The configuration key (e.g., "spark.sql.shuffle.partitions")
- /// * `value` - The configuration value as a string, integer, or boolean
+ /// * `value` - The configuration value, as a string, such as "true" for a
boolean
///
/// # Example
///
/// ```ignore
/// conf.set("spark.sql.shuffle.partitions", "200")?;
- /// conf.set("spark.sql.adaptive.enabled", true)?;
- /// conf.set("spark.sql.maxMetadataStringLength", 100)?;
+ /// conf.set("spark.sql.adaptive.enabled", "true")?;
/// ```
- pub fn set(&self, key: &str, value: &str) -> Result<()> {
+ pub fn set(&self, key: &str, value: &str) -> Result<Vec<String>> {
block_on(self.client.set_config(key, value))
}
/// Set several configuration values in one request, ignoring the failures
to set them,
- /// such as of static configurations, when `silent`. Mirrors
`RuntimeConf._set_all`.
- pub fn set_all(&self, pairs: &[(String, String)], silent: bool) ->
Result<()> {
+ /// such as of static configurations, when `silent`, returning the
warnings of the server.
+ /// Mirrors `RuntimeConf._set_all`.
+ pub fn set_all(&self, pairs: &[(String, String)], silent: bool) ->
Result<Vec<String>> {
block_on(self.client.set_configs(pairs, silent))
}
+ /// Set configuration values in one request, as PySpark's
`RuntimeConf.set` and `_set_all`
+ /// send them: a pair without a value is sent without one, as PySpark
sends None, which the
+ /// server rejects, and with `silent` the failures to set them are
ignored. Returns the
+ /// warnings of the server.
+ pub fn set_pairs(
+ &self,
+ pairs: &[(&str, Option<&str>)],
+ silent: Option<bool>,
+ ) -> Result<Vec<String>> {
+ block_on(self.client.set_config_values(pairs, silent))
+ }
+
/// Get a configuration value by key.
///
/// Returns `Ok(None)` if the key does not exist on the server.
@@ -97,7 +108,8 @@ impl RuntimeConf {
Ok(results)
}
- /// Unset a configuration key (reset to default).
+ /// Unset a configuration key (reset to default), returning the warnings
of the server,
+ /// which PySpark's `RuntimeConf.unset` warns.
///
/// # Arguments
///
@@ -108,7 +120,7 @@ impl RuntimeConf {
/// ```ignore
/// conf.unset("spark.sql.shuffle.partitions")?;
/// ```
- pub fn unset(&self, key: &str) -> Result<()> {
+ pub fn unset(&self, key: &str) -> Result<Vec<String>> {
block_on(self.client.unset_config(key))
}
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index be317f3..6822888 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -274,6 +274,10 @@ def serve(connection):
self.execute_error = None
self.error_details = b""
self.fetched_error_ids = []
+ # The warnings of each Set and Unset, as
ConfigResponse.warnings=3, and the keys
+ # of each Unset.
+ self.config_warnings = []
+ self.unset_confs = []
self.confs = {}
self.read_confs = []
self.set_confs = []
@@ -382,6 +386,10 @@ def serve(connection):
yield header + message_field(13, "r1") + self.execute_payload
yield header + message_field(13, "r2") + message_field(14, b"")
+ def warnings_fields(self):
+ # ConfigResponse.warnings=3, of `config_warnings`.
+ return b"".join(message_field(3, warning) for warning in
self.config_warnings)
+
def fetch_error_details(self, request, context):
# FetchErrorDetailsRequest.error_id=3.
self.fetched_error_ids.append(parse_message(request)[3][0].decode())
@@ -411,9 +419,17 @@ def serve(connection):
pairs = []
for pair in set_request.get(1, []):
pair = parse_message(pair)
- pairs.append((pair[1][0].decode(), pair[2][0].decode()))
+ # A KeyValue of no value=2, as PySpark sends None, is
recorded as None.
+ value = pair.get(2, [None])[0]
+ value = None if value is None else value.decode()
+ pairs.append((pair[1][0].decode(), value))
self.set_confs.append((pairs, set_request.get(2, [None])[0]))
- return message_field(1, request[1][0])
+ return message_field(1, request[1][0]) + self.warnings_fields()
+ if 6 in operation:
+ # Operation.unset=6: Unset.keys=1.
+ keys = parse_message(operation[6][0]).get(1, [])
+ self.unset_confs.append([key.decode() for key in keys])
+ return message_field(1, request[1][0]) + self.warnings_fields()
if 3 in operation:
# Operation.get_with_default=3: GetWithDefault.pairs=1
(KeyValue.key=1,
# value=2), the default of each key that `confs` does not have.
@@ -427,7 +443,8 @@ def serve(connection):
return message_field(1, request[1][0]) + b"".join(pairs)
if 2 not in operation:
context.abort(
- grpc.StatusCode.UNIMPLEMENTED, "only Get, GetWithDefault
and Set are served"
+ grpc.StatusCode.UNIMPLEMENTED,
+ "only Get, GetWithDefault, Set and Unset are served",
)
keys = [key.decode() for key in
parse_message(operation[2][0]).get(1, [])]
self.read_confs.extend(keys)
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 0189f28..d923ff2 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -96,6 +96,31 @@ def
test_udf_and_udtf_calls_take_named_arguments_as_pyspark(session):
assert error.value.getCondition() == "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION"
+def
test_conf_warns_the_warnings_of_the_server_and_takes_its_values_as_pyspark(session):
+ # As PySpark's: set and unset warn the warnings of the server, the server
rejects a set of
+ # None, a set of a float raises a TypeError, and get checks its key and
default.
+ import warnings
+
+ from pyspark.errors import IllegalArgumentException, PySparkTypeError
+
+ key = "spark.sql.execution.arrow.enabled"
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ session.conf.set(key, False)
+ session.conf.unset(key)
+ warning = f"The SQL config '{key}' is NOT supported in Spark Connect"
+ assert [(w.category, str(w.message)) for w in caught] == [(UserWarning,
warning)] * 2
+ with pytest.raises(IllegalArgumentException, match="value cannot be null"):
+ session.conf.set("spark.sql.test.conf.of.none", None)
+ with pytest.raises(TypeError):
+ session.conf.set("spark.sql.test.conf.of.a.float", 1.5)
+ assert session.conf.get("spark.sql.test.conf.of.none", None) is None
+ for args in [(1,), ("spark.sql.test.conf.of.none", 1)]:
+ with pytest.raises(PySparkTypeError) as e:
+ session.conf.get(*args)
+ assert e.value.getCondition() == "NOT_EXPECTED_TYPE"
+
+
def test_conf_get_with_a_none_default(session):
key = "spark.connect.rust.test.unset"
assert session.conf.get(key, None) is None
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index f5f1f5e..83beca1 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -477,6 +477,62 @@ assert SparkSession.getActiveSession() is again and
SparkSession.active() is aga
"""], check=True)
+def
test_conf_warns_the_warnings_of_the_server_and_checks_its_arguments_as_pyspark(mock_connect):
+ # As PySpark's RuntimeConf: set, unset and the options of the builder warn
each warning
+ # of the server; set sends a bool as "true" or "false", an int as its
str(), and None as no
+ # value, and raises the TypeError of protobuf for another value; get raises
+ # NOT_EXPECTED_TYPE of a key, and of a default other than None, that is
not a str.
+ import warnings
+
+ from pyspark.errors import PySparkTypeError
+ from pyspark.sql import SparkSession
+
+ spark, service = mock_connect
+ service.config_warnings = ["one", "two"]
+ with warnings.catch_warnings(record=True) as caught:
+ warnings.simplefilter("always")
+ spark.conf.set("k", True)
+ spark.conf.unset("k")
+ builder =
SparkSession.builder.remote("sc://localhost:1").config("spark.sql.k", "v")
+ assert builder.getOrCreate() is spark
+ assert [(w.category, str(w.message)) for w in caught] == [
+ (UserWarning, "one"),
+ (UserWarning, "two"),
+ ] * 3
+ assert service.unset_confs == [["k"]]
+ service.config_warnings = []
+ for value in [1, "x", None, b"y"]:
+ spark.conf.set("v", value)
+ assert [pairs for pairs, _ in service.set_confs] == [
+ [("k", "true")],
+ [("spark.sql.k", "v")],
+ [("v", "1")],
+ [("v", "x")],
+ [("v", None)],
+ [("v", "y")],
+ ]
+ for value in [1.5, [1], object()]:
+ with pytest.raises(TypeError, match="Cannot set
spark.connect.KeyValue.value"):
+ spark.conf.set("v", value)
+ with pytest.raises(ValueError, match="isn't valid UTF-8"):
+ spark.conf.set("v", b"\xff")
+ assert len(service.set_confs) == 6
+ for args, arg_name, arg_type in [
+ ((1,), "key", "int"),
+ ((None,), "key", "NoneType"),
+ (("k", 1), "default", "int"),
+ (("k", True), "default", "bool"),
+ ]:
+ with pytest.raises(PySparkTypeError) as e:
+ spark.conf.get(*args)
+ assert e.value.getCondition() == "NOT_EXPECTED_TYPE"
+ assert e.value.getMessageParameters() == {
+ "arg_name": arg_name,
+ "expected_type": "str",
+ "arg_type": arg_type,
+ }
+
+
def test_builder_options_are_set_on_the_session_as_pyspark(mock_connect):
# As PySpark's _apply_options: the options of the builder, but
spark.remote and
# spark.master, are set on the session that getOrCreate() returns, also an
existing one,
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]