This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 5b53190f [SPARK-61148][CONNECT][PYTHON] Leave a builder option of None
unset, check the destination of copyFromLocalToFs, and raise NO_ACTIVE_SESSION
for the requests of a stopped session, as PySpark does
5b53190f is described below
commit 5b53190f6c349edc6e0698a6287514994a18613c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:16:10 2026 +0900
[SPARK-61148][CONNECT][PYTHON] Leave a builder option of None unset, check
the destination of copyFromLocalToFs, and raise NO_ACTIVE_SESSION for the
requests of a stopped session, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61148](https://issues.apache.org/jira/browse/SPARK-61148): Leave a
builder option of None unset, check the destination of copyFromLocalToFs, and
raise NO_ACTIVE_SESSION for the requests of a stopped session, as PySpark does.
- Builder options (`config(key, value)`, `config(map=...)`) keep a value of
None, and the options are sent with `_set_all`-style pairs without a value,
silently. The key stays unset on the session, as with PySpark Connect:
`conf.get(key)` raises `SQL_CONF_NOT_FOUND`, `conf.get(key, "unset")` returns
the default, and `getAll` does not have the key. A later value of the same key
replaces an earlier one. Before, the value was stored as `""`.
- `copyFromLocalToFs` raises `AssertionError("destination FS path must be
an absolute path.")` for a relative destination, after the scheme and
local-file checks and before any request, as `_parse_forward_to_fs_artifacts`
does. The core names the artifact `forward_to_fs` + the absolute path
(`forward_to_fs/tmp/x`), not `forward_to_fs//tmp/x`.
- `stop()` releases the session on the server, then closes the core client.
The client is shared by the clones of a session and by its DataFrames, and each
of its requests (execute, analyze, config, interrupt, clone, artifacts, fetch
error details, release execute) raises a `SparkConnectException` of
`NO_ACTIVE_SESSION` once it is closed, as `SparkConnectClient._stub` does.
`ReleaseSession` itself is still sent. A session created afterwards, from the
builder or with `newSession()`, is [...]
- A classed Connect error is raised as `SparkConnectException` (it had been
`PySparkRuntimeError`; no existing error carries a class).
Official PySpark 4.3.0 for the same calls (checked with the reference
client against the same server): `config("spark.r12.none", None)` gives
`SparkNoSuchElementException SQL_CONF_NOT_FOUND`, and the warning "Failed to
set ... to None" is emitted by both; `copyFromLocalToFs("/etc/hosts",
"relative/x")` gives `AssertionError destination FS path must be an absolute
path.`; after `c.stop()`, `c.range(2).count()`, `c.sql(...)`,
`c.conf.get(...)`, `c.version` and `c.cloneSession()` raise ` [...]
### Why are the changes needed?
- `SparkSession.builder.remote(URL).config("spark.x", None).getOrCreate()`
made `spark.conf.get("spark.x")` return `''` and put the key in `getAll`. A
common pattern such as `config("spark.x", os.environ.get("X"))` therefore set
an empty configuration on the session (for example an empty
`spark.sql.session.timeZone`).
- `copyFromLocalToFs(local, "relative/out.txt")` reached the server and
failed there with `UnsupportedOperationException`, instead of the
`AssertionError` that PySpark raises before any request. An absolute
destination was named `forward_to_fs//tmp/x`.
- After `stop()`, a session kept running its requests on the server and
silently reused the released session ID (for example `range(2).count()`
returned 2, and `version` returned `4.3.0`), where PySpark raises
`NO_ACTIVE_SESSION`.
### Does this PR introduce _any_ user-facing change?
Yes.
- A builder option of None leaves its key unset instead of setting it to
`""`.
- A relative `copyFromLocalToFs` destination raises `AssertionError`
(`destination FS path must be an absolute path.`) before any request.
- A request of a stopped session, or of a DataFrame of it, raises
`SparkConnectException` with condition `NO_ACTIVE_SESSION`.
- A classed Connect error is a `SparkConnectException` instead of a
`PySparkRuntimeError`.
### How was this patch tested?
- `test_copy_from_local_to_fs_requires_an_absolute_destination` (offline):
the relative destination raises the AssertionError before any request; same as
official PySpark 4.3.0.
- `test_requests_of_a_stopped_session_raise_no_active_session_as_pyspark`
(offline, no server): `count`, `sql().collect()`, `conf.get` and `version` of a
stopped `newSession()` raise `SparkConnectException` `NO_ACTIVE_SESSION`; same
as official PySpark 4.3.0.
- `test_builder_config_none_leaves_the_key_unset_as_pyspark` (live):
`config(key, None)`, `config(map={key: None})`; the key is unset
(`SQL_CONF_NOT_FOUND`, not in `getAll`), the neighbouring option is set; same
as official PySpark 4.3.0.
- `test_stopped_session_raises_no_active_session_while_the_others_work`
(live): a stopped `newSession()` raises `NO_ACTIVE_SESSION`, and the main
session still works; same as official PySpark 4.3.0.
- Rust unit tests in `crates/spark-connect/src/session.rs`:
`requests_of_a_stopped_session_raise_no_active_session` and
`forward_to_fs_artifact_name_is_the_prefix_and_the_absolute_path`.
- The tests of both crates, e2e included, against a local Spark 4.3.0
server (877 passed), the build of all tests, also with `wasm-udf`, the offline
Python suites (2203 passed, 8 skipped), and the live drop-in, Arrow and ML
suites (186 passed) and live cases (10 passed). rustfmt passes, and clippy
reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #338 from
HyukjinKwon/SPARK-61148-builder-none-copy-dest-stopped-session.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/errors.rs | 10 +++++-
crates/pyspark-rs/src/session.rs | 61 ++++++++++++++++++++++++---------
crates/spark-connect-core/src/client.rs | 46 +++++++++++++++++++++++++
crates/spark-connect/src/session.rs | 44 +++++++++++++++++++++---
python/tests/test_arrow_handoff.py | 3 +-
python/tests/test_dropin_e2e.py | 30 ++++++++++++++++
python/tests/test_dropin_offline.py | 38 ++++++++++++++++++--
7 files changed, 208 insertions(+), 24 deletions(-)
diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index 23b34746..b9a17f7c 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -217,8 +217,16 @@ pub fn spark_error_to_py_exception(err: SparkError) ->
PyErr {
}
};
+ // A classed Connect error, such as `NO_ACTIVE_SESSION`, is a
`SparkConnectException`, as
+ // the reference client raises it.
+ let exc_class = if err.kind == SparkErrorKind::Connect &&
!err.error_class.is_empty() {
+ py.import("pyspark.errors.exceptions.connect")
+ .and_then(|module| module.getattr("SparkConnectException"))
+ } else {
+ errors_module.getattr(class_name)
+ };
// Get the exception class from the module
- match errors_module.getattr(class_name) {
+ match exc_class {
Ok(exc_class) => {
// Instantiate the exception class: of the error class and
message parameters
// of an error of a PySpark error class, otherwise of the
message.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1d822455..ba5aa38d 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -24,7 +24,7 @@ use crate::streaming::{PyDataStreamReader,
PyStreamingQueryManager};
pub struct PySparkSessionBuilder {
remote_url: Option<String>,
/// Runtime configs set via `.config(...)`, applied after the session
connects.
- configs: Vec<(String, String)>,
+ configs: Vec<(String, Option<String>)>,
/// The hook factories of `_registerHook`, which a new session calls with
itself, as
/// PySpark's builder does.
hook_factories: Vec<std::sync::Arc<Py<PyAny>>>,
@@ -104,16 +104,16 @@ impl PySparkSessionBuilder {
let py = slf.py();
if let Some(k) = key {
let v = match value {
- Some(v) => crate::coerce_option_value(v)?.unwrap_or_default(),
- None => String::new(),
+ Some(v) => crate::coerce_option_value(v)?,
+ None => None,
};
- slf.set_option(py, k, v)?;
+ slf.set_option_value(py, k, v)?;
}
if let Some(m) = map {
for (k, v) in m.iter() {
let ks = k.str()?.to_string();
- let vs = crate::coerce_option_value(&v)?.unwrap_or_default();
- slf.set_option(py, &ks, vs)?;
+ let vs = crate::coerce_option_value(&v)?;
+ slf.set_option_value(py, &ks, vs)?;
}
}
if let Some(c) = conf {
@@ -278,8 +278,19 @@ impl PySparkSessionBuilder {
/// Set an option, capturing `spark.remote` as the endpoint, and validate
the startup URLs
/// after it, as PySpark's `config`.
fn set_option(&mut self, py: Python<'_>, key: &str, value: String) ->
PyResult<()> {
+ self.set_option_value(py, key, Some(value))
+ }
+
+ /// Set an option whose value may be None, which PySpark keeps in its
options: such a key is
+ /// not set on the session, as `_set_all` sends it without a value.
+ fn set_option_value(
+ &mut self,
+ py: Python<'_>,
+ key: &str,
+ value: Option<String>,
+ ) -> PyResult<()> {
if key == "spark.remote" {
- self.remote_url = Some(value);
+ self.remote_url = value;
} else {
self.configs.push((key.to_string(), value));
}
@@ -296,7 +307,7 @@ impl PySparkSessionBuilder {
.iter()
.rev()
.find(|(key, _)| key == "spark.master")
- .map(|(_, value)| value.clone())
+ .and_then(|(_, value)| value.clone())
.or_else(|| std::env::var("MASTER").ok());
let remote = self
.remote_url
@@ -351,18 +362,24 @@ impl PySparkSessionBuilder {
// SparkError is large, as the core crate allows for its API.
#[allow(clippy::result_large_err)]
fn apply_options(&self, py: Python<'_>, session: &SparkSession) ->
PyResult<()> {
- let options: Vec<(String, String)> = self
- .configs
- .iter()
- .filter(|(key, _)| key != "spark.master")
- .cloned()
- .collect();
+ // A later value of a key replaces an earlier one, as the options of
PySpark's builder do.
+ let mut options: Vec<(String, Option<String>)> = Vec::new();
+ for (key, value) in self.configs.iter().filter(|(key, _)| key !=
"spark.master") {
+ match options.iter_mut().find(|(k, _)| k == key) {
+ Some(entry) => entry.1 = value.clone(),
+ None => options.push((key.clone(), value.clone())),
+ }
+ }
if options.is_empty() {
return Ok(());
}
+ let pairs: Vec<(&str, Option<&str>)> = options
+ .iter()
+ .map(|(key, value)| (key.as_str(), value.as_deref()))
+ .collect();
let conf = session.conf();
let warnings = py
- .detach(|| conf.set_all(&options, true).map_err(Box::new))
+ .detach(|| conf.set_pairs(&pairs, Some(true)).map_err(Box::new))
.to_pyerr()?;
// As PySpark's `_set_all`, which warns each warning of the server.
crate::conf::warn_all(py, &warnings)
@@ -1663,7 +1680,8 @@ impl PySparkSession {
/// Copy a local file to the Spark-managed filesystem
(`copyFromLocalToFs`), with the errors
/// of PySpark Connect's: `NO_SCHEMA_AND_DRIVER_DEFAULT_SCHEME` for a
destination of a scheme,
- /// and an `AssertionError` for a local path that is not a file.
+ /// an `AssertionError` for a local path that is not a file, and for a
destination that is
+ /// not an absolute path.
#[pyo3(name = "copyFromLocalToFs")]
fn copy_from_local_to_fs(
&self,
@@ -1696,6 +1714,17 @@ impl PySparkSession {
"local path must be a file path.",
));
}
+ let is_absolute: bool = py
+ .import("pathlib")?
+ .getattr("Path")?
+ .call1((dest_path,))?
+ .call_method0("is_absolute")?
+ .extract()?;
+ if !is_absolute {
+ return Err(pyo3::exceptions::PyAssertionError::new_err(
+ "destination FS path must be an absolute path.",
+ ));
+ }
py.detach(|| self.session.copy_from_local_to_fs(local_path, dest_path))
.to_pyerr()
}
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 3552b86f..a4d0a6a7 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -238,6 +238,9 @@ pub struct SparkConnectClient {
/// The hook of the executions, if any, shared by the clients of the
clones of the session,
/// as PySpark's clones keep the hooks of a session.
execute_plan_hook:
std::sync::Arc<std::sync::RwLock<Option<ExecutePlanHook>>>,
+ /// Whether the client is closed, shared by its clones: its RPCs then raise
+ /// `NO_ACTIVE_SESSION`, as PySpark's `SparkConnectClient._stub` does once
it is closed.
+ closed: std::sync::Arc<std::sync::atomic::AtomicBool>,
}
/// The user context extensions of a client, as PySpark's `SparkConnectClient`
keeps them: those
@@ -338,6 +341,7 @@ impl SparkConnectClient {
retry_policy: RetryPolicy::default(),
rpc_deadlines: RpcDeadlines::default(),
execute_plan_hook: Default::default(),
+ closed: Default::default(),
})
}
@@ -363,9 +367,35 @@ impl SparkConnectClient {
retry_policy: self.retry_policy.clone(),
rpc_deadlines: self.rpc_deadlines.clone(),
execute_plan_hook: std::sync::Arc::clone(&self.execute_plan_hook),
+ closed: Default::default(),
}
}
+ /// Close this client, as PySpark's `SparkConnectClient.close` closes its
channel: its RPCs
+ /// raise `NO_ACTIVE_SESSION` from now on, and its clones with them.
`release_session` is
+ /// still sent, as `stop()` sends it before it closes the client.
+ pub fn close(&self) {
+ self.closed.store(true, std::sync::atomic::Ordering::SeqCst);
+ }
+
+ /// Whether the client is closed. See [`SparkConnectClient::close`].
+ pub fn is_closed(&self) -> bool {
+ self.closed.load(std::sync::atomic::Ordering::SeqCst)
+ }
+
+ /// The error of an RPC of a closed client, as PySpark's
`SparkConnectClient._stub` raises it.
+ #[allow(clippy::result_large_err)] // SparkError is large; changing the
type would break the API
+ fn ensure_open(&self) -> Result<()> {
+ if self.is_closed() {
+ return Err(SparkError::classed(
+ crate::error::SparkErrorKind::Connect,
+ "NO_ACTIVE_SESSION",
+ &[],
+ ));
+ }
+ Ok(())
+ }
+
/// Set the hook of the executions of this client and of its clones, or
clear it.
pub fn set_execute_plan_hook(&self, hook: Option<ExecutePlanHook>) {
*self
@@ -521,6 +551,7 @@ impl SparkConnectClient {
&self,
mut request: ExecutePlanRequest,
) -> Result<Streaming<ExecutePlanResponse>> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
let request = self.hooked(request)?;
@@ -538,6 +569,7 @@ impl SparkConnectClient {
/// and deep plans don't hit the recursion limit - the server sees exactly
the bytes
/// the reference client built.
pub async fn execute_plan_raw(&self, request: Vec<u8>) ->
Result<Streaming<Vec<u8>>> {
+ self.ensure_open()?;
let mut grpc = tonic::client::Grpc::new(self.channel.clone())
.max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
.max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -556,6 +588,7 @@ impl SparkConnectClient {
/// ReattachExecute, byte-exact passthrough returning a raw response
stream.
pub async fn reattach_execute_raw(&self, request: Vec<u8>) ->
Result<Streaming<Vec<u8>>> {
+ self.ensure_open()?;
let mut grpc = tonic::client::Grpc::new(self.channel.clone())
.max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
.max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -579,6 +612,7 @@ impl SparkConnectClient {
&self,
mut request: ReattachExecuteRequest,
) -> Result<Streaming<ExecutePlanResponse>> {
+ self.ensure_open()?;
self.fill_user_context(&mut request.user_context);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
@@ -595,6 +629,7 @@ impl SparkConnectClient {
&self,
mut request: ReleaseExecuteRequest,
) -> Result<ReleaseExecuteResponse> {
+ self.ensure_open()?;
self.fill_user_context(&mut request.user_context);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
@@ -612,6 +647,7 @@ impl SparkConnectClient {
/// returns the raw response bytes, avoiding the client-side limit
(matching the
/// reference client, which never re-decodes the request it built).
pub async fn analyze_plan_raw(&self, request: Vec<u8>) -> Result<Vec<u8>> {
+ self.ensure_open()?;
let mut grpc = tonic::client::Grpc::new(self.channel.clone())
.max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
.max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -638,6 +674,7 @@ impl SparkConnectClient {
&self,
mut request: AnalyzePlanRequest,
) -> Result<AnalyzePlanResponse> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
@@ -691,6 +728,7 @@ impl SparkConnectClient {
&self,
mut request: ExecutePlanRequest,
) -> Result<ReattachableResponseStream> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
// Before the iterator: its ReattachExecute and ReleaseExecute
requests take the
// user context of this request, and the request of the hook, as
PySpark's iterator
@@ -748,6 +786,7 @@ impl SparkConnectClient {
&self,
mut request: FetchErrorDetailsRequest,
) -> Result<FetchErrorDetailsResponse> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
let deadline = self.rpc_deadlines.fetch_error_details;
@@ -842,6 +881,7 @@ impl SparkConnectClient {
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.config`.
pub async fn config(&self, mut request: ConfigRequest) ->
Result<ConfigResponse> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
@@ -862,6 +902,7 @@ impl SparkConnectClient {
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.interrupt`.
pub async fn interrupt(&self, mut request: InterruptRequest) ->
Result<InterruptResponse> {
+ self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
@@ -1167,6 +1208,7 @@ impl SparkConnectClient {
/// and catalog state, into a new session, of the id `new_session_id` if
one is given. The
/// returned client, of the same channel, is of that session.
pub async fn clone_session(&self, new_session_id: Option<&str>) ->
Result<Self> {
+ self.ensure_open()?;
let request = CloneSessionRequest {
session_id: self.session_id.clone(),
user_context: Some(UserContext {
@@ -1232,6 +1274,7 @@ impl SparkConnectClient {
archive: bool,
file: bool,
) -> Result<()> {
+ self.ensure_open()?;
// Each path is named as PySpark's ArtifactManager names it, by the
flags and its file
// extension (see `parse_artifact`).
for path in paths {
@@ -1249,6 +1292,7 @@ impl SparkConnectClient {
/// Used by `SparkSession::copy_from_local_to_fs`, which names the artifact
/// `forward_to_fs/<dest>` so the server writes it to the target
filesystem path.
pub async fn add_named_artifact(&self, name: &str, local_path: &str) ->
Result<()> {
+ self.ensure_open()?;
let artifact =
NamedArtifact::new(name.to_string(),
Box::new(FileArtifact::new(local_path)));
self.upload_artifact(&artifact).await
@@ -1260,6 +1304,7 @@ impl SparkConnectClient {
/// As `SparkConnectClient.cache_artifacts`, the status check and the
upload are one
/// attempt under the retry policy, so a transient failure of either
retries both.
pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) ->
Result<Vec<String>> {
+ self.ensure_open()?;
use bytes::Bytes;
use std::collections::HashSet;
@@ -1383,6 +1428,7 @@ impl SparkConnectClient {
&self,
names: &[&str],
) -> Result<std::collections::HashMap<String, bool>> {
+ self.ensure_open()?;
let names: Vec<String> = names.iter().map(|n| n.to_string()).collect();
match self.artifact_status_rpc(&names).await {
Ok(statuses) => Ok(statuses),
diff --git a/crates/spark-connect/src/session.rs
b/crates/spark-connect/src/session.rs
index 9343f736..83ce8898 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -903,10 +903,13 @@ impl SparkSession {
/// Copy a local file to the driver's filesystem at `dest_path`.
///
/// Mirrors `pyspark.sql.SparkSession.copyFromLocalToFs`: uploads the file
as a
- /// `forward_to_fs/<dest_path>` artifact, which the server writes to
`dest_path`.
+ /// `forward_to_fs<dest_path>` artifact, which the server writes to
`dest_path`. As the
+ /// reference, `dest_path` is an absolute path, so the name is the prefix
and the path.
pub fn copy_from_local_to_fs(&self, local_path: &str, dest_path: &str) ->
Result<()> {
- let name = format!("forward_to_fs/{}", dest_path);
- block_on(self.client.add_named_artifact(&name, local_path))
+ block_on(
+ self.client
+ .add_named_artifact(&forward_to_fs_artifact_name(dest_path),
local_path),
+ )
}
/// Register a user-defined function on the session so it can be
referenced by
@@ -1026,7 +1029,11 @@ impl SparkSession {
if self.stopped.swap(true, Ordering::SeqCst) {
return Ok(());
}
- block_on(self.client.release_session())
+ // As PySpark's `stop()`, the client is closed after the session is
released, so that
+ // its requests raise `NO_ACTIVE_SESSION` from now on.
+ let released = block_on(self.client.release_session());
+ self.client.close();
+ released
}
/// Release the session on the server, as
`SparkConnectClient.release_session`: its state
@@ -1039,6 +1046,7 @@ impl SparkSession {
/// closes its channel: it is stopped, and `stop()` does nothing more.
pub fn close(&self) {
self.stopped.store(true, Ordering::SeqCst);
+ self.client.close();
}
/// Whether `stop()` has been called on this session. Mirrors
@@ -1048,6 +1056,12 @@ impl SparkSession {
}
}
+/// The name of the artifact that copies a file to `dest_path`, as the
reference's
+/// `FORWARD_TO_FS_PREFIX + dest_path`: the prefix is followed by the absolute
path itself.
+fn forward_to_fs_artifact_name(dest_path: &str) -> String {
+ format!("forward_to_fs{}", dest_path)
+}
+
impl Clone for SparkSession {
fn clone(&self) -> Self {
SparkSession {
@@ -2618,6 +2632,28 @@ mod tests {
assert!(spark.stop().is_ok());
}
+ #[test]
+ fn requests_of_a_stopped_session_raise_no_active_session() {
+ // Port 1: nothing listens, so a request that reached the wire would
fail differently.
+ // As PySpark's closed client, the stopped session raises before it
sends anything.
+ let spark = SparkSession::builder()
+ .remote("sc://localhost:1")
+ .get_or_create()
+ .expect("failed to build session");
+ let _ = spark.stop();
+ let err = spark
+ .conf()
+ .get("spark.sql.shuffle.partitions")
+ .expect_err("a request of a stopped session must fail");
+ assert_eq!(err.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+ }
+
+ #[test]
+ fn forward_to_fs_artifact_name_is_the_prefix_and_the_absolute_path() {
+ // As the reference's `FORWARD_TO_FS_PREFIX + dest_path`, with no
separator added.
+ assert_eq!(forward_to_fs_artifact_name("/tmp/x"),
"forward_to_fs/tmp/x");
+ }
+
#[test]
fn version_cache_follows_session_identity() {
let spark = session();
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index a26457d5..60563cd1 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -760,7 +760,8 @@ def test_file_upload_attribution_on_every_frame(
destinations = ["/wire/small.txt", "/wire/large.txt"]
for path, destination in zip([small, large], destinations):
spark.copyFromLocalToFs(str(path), destination)
- artifact_names = ["forward_to_fs/" + destination for destination
in destinations]
+ # As PySpark's FORWARD_TO_FS_PREFIX + dest_path: the path is
absolute, so no separator.
+ artifact_names = ["forward_to_fs" + destination for destination in
destinations]
payloads = {context["payload"] for context in state["upload_contexts"]}
assert payloads == {"batch", "begin_chunk", "chunk"}, (
f"Expected all artifact frame kinds, got {payloads}. "
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 5972870e..4afd6911 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2852,6 +2852,36 @@ def
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(sessio
failed.stop()
+def test_builder_config_none_leaves_the_key_unset_as_pyspark(session):
+ # As PySpark's `_set_all`: an option of None, of config or of map, leaves
its key unset on
+ # the session, while the other options of the builder are set.
+ built = (SparkSession.builder.remote(os.environ["SPARK_REMOTE"])
+ .config("spark.r12.none", None)
+ .config("spark.r12.kept", "v")
+ .config(map={"spark.r12.map": None})
+ .getOrCreate())
+ assert built.session_id == session.session_id
+ assert session.conf.get("spark.r12.kept") == "v"
+ assert session.conf.get("spark.r12.none", "unset") == "unset"
+ assert session.conf.get("spark.r12.map", "unset") == "unset"
+ assert "spark.r12.none" not in session.conf.getAll
+ with pytest.raises(Exception) as e:
+ session.conf.get("spark.r12.none")
+ assert e.value.getCondition() == "SQL_CONF_NOT_FOUND"
+
+
+def
test_stopped_session_raises_no_active_session_while_the_others_work(session):
+ from pyspark.errors.exceptions.connect import SparkConnectException
+
+ other = session.newSession()
+ other.stop()
+ assert other.is_stopped
+ with pytest.raises(SparkConnectException) as e:
+ other.range(2).count()
+ assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+ assert session.range(3).count() == 3
+
+
def
test_stream_writer_starts_a_query_of_its_real_time_trigger_as_pyspark(session):
# As PySpark's, the real-time trigger reaches the server, which runs
real-time mode for
# some sinks only, not the memory sink.
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index 2ce2d6d4..08f61f51 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -11052,6 +11052,38 @@ def
test_positional_indexing_integer_subclass_and_string_lookup(positional_index
assert service.calls == before
+def test_copy_from_local_to_fs_requires_an_absolute_destination(mock_connect,
tmp_path):
+ # As PySpark Connect's `_parse_forward_to_fs_artifacts`: the destination
must be an absolute
+ # path, which is checked before any request, so the server never sees a
relative one.
+ spark, _ = mock_connect
+ local = tmp_path / "local.txt"
+ local.write_text("x")
+ with pytest.raises(AssertionError, match="destination FS path must be an
absolute path."):
+ spark.copyFromLocalToFs(str(local), "relative/out.txt")
+
+
+def test_requests_of_a_stopped_session_raise_no_active_session_as_pyspark():
+ # As PySpark's closed client: stop() releases the session and closes the
client, so each
+ # request of the stopped session, and of its DataFrames, raises
NO_ACTIVE_SESSION.
+ from pyspark.errors.exceptions.connect import SparkConnectException
+
+ spark = _offline_session()
+ other = spark.newSession()
+ other.stop()
+ assert other.is_stopped
+ requests = [
+ lambda: other.range(2).count(),
+ lambda: other.sql("select 1").collect(),
+ lambda: other.conf.get("spark.sql.shuffle.partitions"),
+ lambda: other.version,
+ ]
+ for request in requests:
+ with pytest.raises(SparkConnectException) as e:
+ request()
+ assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+ spark.stop()
+
+
def
test_positional_indexing_invalid_index_types_do_not_analyze(positional_indexing_analyzed):
from pyspark.errors import PySparkTypeError
@@ -12671,13 +12703,15 @@ def
test_version_cache_active_session_handle_shares_cache(version_cache_session)
def
test_version_cache_stop_before_first_lookup_keeps_rpc_behavior(version_cache_session):
+ # As PySpark's closed client, a lookup after stop() raises
NO_ACTIVE_SESSION without a request.
spark, service = version_cache_session
spark.stop()
assert service.calls == 0
service.failure = "rpc"
- with pytest.raises(Exception):
+ with pytest.raises(Exception) as e:
_ = spark.version
- assert service.calls == 1
+ assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+ assert service.calls == 0
# ---------------------------------------------------------------------------
printSchema
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]