This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new 5b53190f [SPARK-61148][CONNECT][PYTHON] Leave a builder option of None 
unset, check the destination of copyFromLocalToFs, and raise NO_ACTIVE_SESSION 
for the requests of a stopped session, as PySpark does
5b53190f is described below

commit 5b53190f6c349edc6e0698a6287514994a18613c
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 12:16:10 2026 +0900

    [SPARK-61148][CONNECT][PYTHON] Leave a builder option of None unset, check 
the destination of copyFromLocalToFs, and raise NO_ACTIVE_SESSION for the 
requests of a stopped session, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61148](https://issues.apache.org/jira/browse/SPARK-61148): Leave a 
builder option of None unset, check the destination of copyFromLocalToFs, and 
raise NO_ACTIVE_SESSION for the requests of a stopped session, as PySpark does.
    
    - Builder options (`config(key, value)`, `config(map=...)`) keep a value of 
None, and the options are sent with `_set_all`-style pairs without a value, 
silently. The key stays unset on the session, as with PySpark Connect: 
`conf.get(key)` raises `SQL_CONF_NOT_FOUND`, `conf.get(key, "unset")` returns 
the default, and `getAll` does not have the key. A later value of the same key 
replaces an earlier one. Before, the value was stored as `""`.
    - `copyFromLocalToFs` raises `AssertionError("destination FS path must be 
an absolute path.")` for a relative destination, after the scheme and 
local-file checks and before any request, as `_parse_forward_to_fs_artifacts` 
does. The core names the artifact `forward_to_fs` + the absolute path 
(`forward_to_fs/tmp/x`), not `forward_to_fs//tmp/x`.
    - `stop()` releases the session on the server, then closes the core client. 
The client is shared by the clones of a session and by its DataFrames, and each 
of its requests (execute, analyze, config, interrupt, clone, artifacts, fetch 
error details, release execute) raises a `SparkConnectException` of 
`NO_ACTIVE_SESSION` once it is closed, as `SparkConnectClient._stub` does. 
`ReleaseSession` itself is still sent. A session created afterwards, from the 
builder or with `newSession()`, is [...]
    - A classed Connect error is raised as `SparkConnectException` (it had been 
`PySparkRuntimeError`; no existing error carries a class).
    
    Official PySpark 4.3.0 for the same calls (checked with the reference 
client against the same server): `config("spark.r12.none", None)` gives 
`SparkNoSuchElementException SQL_CONF_NOT_FOUND`, and the warning "Failed to 
set ... to None" is emitted by both; `copyFromLocalToFs("/etc/hosts", 
"relative/x")` gives `AssertionError destination FS path must be an absolute 
path.`; after `c.stop()`, `c.range(2).count()`, `c.sql(...)`, 
`c.conf.get(...)`, `c.version` and `c.cloneSession()` raise ` [...]
    
    ### Why are the changes needed?
    
    - `SparkSession.builder.remote(URL).config("spark.x", None).getOrCreate()` 
made `spark.conf.get("spark.x")` return `''` and put the key in `getAll`. A 
common pattern such as `config("spark.x", os.environ.get("X"))` therefore set 
an empty configuration on the session (for example an empty 
`spark.sql.session.timeZone`).
    - `copyFromLocalToFs(local, "relative/out.txt")` reached the server and 
failed there with `UnsupportedOperationException`, instead of the 
`AssertionError` that PySpark raises before any request. An absolute 
destination was named `forward_to_fs//tmp/x`.
    - After `stop()`, a session kept running its requests on the server and 
silently reused the released session ID (for example `range(2).count()` 
returned 2, and `version` returned `4.3.0`), where PySpark raises 
`NO_ACTIVE_SESSION`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes.
    
    - A builder option of None leaves its key unset instead of setting it to 
`""`.
    - A relative `copyFromLocalToFs` destination raises `AssertionError` 
(`destination FS path must be an absolute path.`) before any request.
    - A request of a stopped session, or of a DataFrame of it, raises 
`SparkConnectException` with condition `NO_ACTIVE_SESSION`.
    - A classed Connect error is a `SparkConnectException` instead of a 
`PySparkRuntimeError`.
    
    ### How was this patch tested?
    
    - `test_copy_from_local_to_fs_requires_an_absolute_destination` (offline): 
the relative destination raises the AssertionError before any request; same as 
official PySpark 4.3.0.
    - `test_requests_of_a_stopped_session_raise_no_active_session_as_pyspark` 
(offline, no server): `count`, `sql().collect()`, `conf.get` and `version` of a 
stopped `newSession()` raise `SparkConnectException` `NO_ACTIVE_SESSION`; same 
as official PySpark 4.3.0.
    - `test_builder_config_none_leaves_the_key_unset_as_pyspark` (live): 
`config(key, None)`, `config(map={key: None})`; the key is unset 
(`SQL_CONF_NOT_FOUND`, not in `getAll`), the neighbouring option is set; same 
as official PySpark 4.3.0.
    - `test_stopped_session_raises_no_active_session_while_the_others_work` 
(live): a stopped `newSession()` raises `NO_ACTIVE_SESSION`, and the main 
session still works; same as official PySpark 4.3.0.
    - Rust unit tests in `crates/spark-connect/src/session.rs`: 
`requests_of_a_stopped_session_raise_no_active_session` and 
`forward_to_fs_artifact_name_is_the_prefix_and_the_absolute_path`.
    - The tests of both crates, e2e included, against a local Spark 4.3.0 
server (877 passed), the build of all tests, also with `wasm-udf`, the offline 
Python suites (2203 passed, 8 skipped), and the live drop-in, Arrow and ML 
suites (186 passed) and live cases (10 passed). rustfmt passes, and clippy 
reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #338 from 
HyukjinKwon/SPARK-61148-builder-none-copy-dest-stopped-session.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/errors.rs         | 10 +++++-
 crates/pyspark-rs/src/session.rs        | 61 ++++++++++++++++++++++++---------
 crates/spark-connect-core/src/client.rs | 46 +++++++++++++++++++++++++
 crates/spark-connect/src/session.rs     | 44 +++++++++++++++++++++---
 python/tests/test_arrow_handoff.py      |  3 +-
 python/tests/test_dropin_e2e.py         | 30 ++++++++++++++++
 python/tests/test_dropin_offline.py     | 38 ++++++++++++++++++--
 7 files changed, 208 insertions(+), 24 deletions(-)

diff --git a/crates/pyspark-rs/src/errors.rs b/crates/pyspark-rs/src/errors.rs
index 23b34746..b9a17f7c 100644
--- a/crates/pyspark-rs/src/errors.rs
+++ b/crates/pyspark-rs/src/errors.rs
@@ -217,8 +217,16 @@ pub fn spark_error_to_py_exception(err: SparkError) -> 
PyErr {
             }
         };
 
+        // A classed Connect error, such as `NO_ACTIVE_SESSION`, is a 
`SparkConnectException`, as
+        // the reference client raises it.
+        let exc_class = if err.kind == SparkErrorKind::Connect && 
!err.error_class.is_empty() {
+            py.import("pyspark.errors.exceptions.connect")
+                .and_then(|module| module.getattr("SparkConnectException"))
+        } else {
+            errors_module.getattr(class_name)
+        };
         // Get the exception class from the module
-        match errors_module.getattr(class_name) {
+        match exc_class {
             Ok(exc_class) => {
                 // Instantiate the exception class: of the error class and 
message parameters
                 // of an error of a PySpark error class, otherwise of the 
message.
diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 1d822455..ba5aa38d 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -24,7 +24,7 @@ use crate::streaming::{PyDataStreamReader, 
PyStreamingQueryManager};
 pub struct PySparkSessionBuilder {
     remote_url: Option<String>,
     /// Runtime configs set via `.config(...)`, applied after the session 
connects.
-    configs: Vec<(String, String)>,
+    configs: Vec<(String, Option<String>)>,
     /// The hook factories of `_registerHook`, which a new session calls with 
itself, as
     /// PySpark's builder does.
     hook_factories: Vec<std::sync::Arc<Py<PyAny>>>,
@@ -104,16 +104,16 @@ impl PySparkSessionBuilder {
         let py = slf.py();
         if let Some(k) = key {
             let v = match value {
-                Some(v) => crate::coerce_option_value(v)?.unwrap_or_default(),
-                None => String::new(),
+                Some(v) => crate::coerce_option_value(v)?,
+                None => None,
             };
-            slf.set_option(py, k, v)?;
+            slf.set_option_value(py, k, v)?;
         }
         if let Some(m) = map {
             for (k, v) in m.iter() {
                 let ks = k.str()?.to_string();
-                let vs = crate::coerce_option_value(&v)?.unwrap_or_default();
-                slf.set_option(py, &ks, vs)?;
+                let vs = crate::coerce_option_value(&v)?;
+                slf.set_option_value(py, &ks, vs)?;
             }
         }
         if let Some(c) = conf {
@@ -278,8 +278,19 @@ impl PySparkSessionBuilder {
     /// Set an option, capturing `spark.remote` as the endpoint, and validate 
the startup URLs
     /// after it, as PySpark's `config`.
     fn set_option(&mut self, py: Python<'_>, key: &str, value: String) -> 
PyResult<()> {
+        self.set_option_value(py, key, Some(value))
+    }
+
+    /// Set an option whose value may be None, which PySpark keeps in its 
options: such a key is
+    /// not set on the session, as `_set_all` sends it without a value.
+    fn set_option_value(
+        &mut self,
+        py: Python<'_>,
+        key: &str,
+        value: Option<String>,
+    ) -> PyResult<()> {
         if key == "spark.remote" {
-            self.remote_url = Some(value);
+            self.remote_url = value;
         } else {
             self.configs.push((key.to_string(), value));
         }
@@ -296,7 +307,7 @@ impl PySparkSessionBuilder {
             .iter()
             .rev()
             .find(|(key, _)| key == "spark.master")
-            .map(|(_, value)| value.clone())
+            .and_then(|(_, value)| value.clone())
             .or_else(|| std::env::var("MASTER").ok());
         let remote = self
             .remote_url
@@ -351,18 +362,24 @@ impl PySparkSessionBuilder {
     // SparkError is large, as the core crate allows for its API.
     #[allow(clippy::result_large_err)]
     fn apply_options(&self, py: Python<'_>, session: &SparkSession) -> 
PyResult<()> {
-        let options: Vec<(String, String)> = self
-            .configs
-            .iter()
-            .filter(|(key, _)| key != "spark.master")
-            .cloned()
-            .collect();
+        // A later value of a key replaces an earlier one, as the options of 
PySpark's builder do.
+        let mut options: Vec<(String, Option<String>)> = Vec::new();
+        for (key, value) in self.configs.iter().filter(|(key, _)| key != 
"spark.master") {
+            match options.iter_mut().find(|(k, _)| k == key) {
+                Some(entry) => entry.1 = value.clone(),
+                None => options.push((key.clone(), value.clone())),
+            }
+        }
         if options.is_empty() {
             return Ok(());
         }
+        let pairs: Vec<(&str, Option<&str>)> = options
+            .iter()
+            .map(|(key, value)| (key.as_str(), value.as_deref()))
+            .collect();
         let conf = session.conf();
         let warnings = py
-            .detach(|| conf.set_all(&options, true).map_err(Box::new))
+            .detach(|| conf.set_pairs(&pairs, Some(true)).map_err(Box::new))
             .to_pyerr()?;
         // As PySpark's `_set_all`, which warns each warning of the server.
         crate::conf::warn_all(py, &warnings)
@@ -1663,7 +1680,8 @@ impl PySparkSession {
 
     /// Copy a local file to the Spark-managed filesystem 
(`copyFromLocalToFs`), with the errors
     /// of PySpark Connect's: `NO_SCHEMA_AND_DRIVER_DEFAULT_SCHEME` for a 
destination of a scheme,
-    /// and an `AssertionError` for a local path that is not a file.
+    /// an `AssertionError` for a local path that is not a file, and for a 
destination that is
+    /// not an absolute path.
     #[pyo3(name = "copyFromLocalToFs")]
     fn copy_from_local_to_fs(
         &self,
@@ -1696,6 +1714,17 @@ impl PySparkSession {
                 "local path must be a file path.",
             ));
         }
+        let is_absolute: bool = py
+            .import("pathlib")?
+            .getattr("Path")?
+            .call1((dest_path,))?
+            .call_method0("is_absolute")?
+            .extract()?;
+        if !is_absolute {
+            return Err(pyo3::exceptions::PyAssertionError::new_err(
+                "destination FS path must be an absolute path.",
+            ));
+        }
         py.detach(|| self.session.copy_from_local_to_fs(local_path, dest_path))
             .to_pyerr()
     }
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 3552b86f..a4d0a6a7 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -238,6 +238,9 @@ pub struct SparkConnectClient {
     /// The hook of the executions, if any, shared by the clients of the 
clones of the session,
     /// as PySpark's clones keep the hooks of a session.
     execute_plan_hook: 
std::sync::Arc<std::sync::RwLock<Option<ExecutePlanHook>>>,
+    /// Whether the client is closed, shared by its clones: its RPCs then raise
+    /// `NO_ACTIVE_SESSION`, as PySpark's `SparkConnectClient._stub` does once 
it is closed.
+    closed: std::sync::Arc<std::sync::atomic::AtomicBool>,
 }
 
 /// The user context extensions of a client, as PySpark's `SparkConnectClient` 
keeps them: those
@@ -338,6 +341,7 @@ impl SparkConnectClient {
             retry_policy: RetryPolicy::default(),
             rpc_deadlines: RpcDeadlines::default(),
             execute_plan_hook: Default::default(),
+            closed: Default::default(),
         })
     }
 
@@ -363,9 +367,35 @@ impl SparkConnectClient {
             retry_policy: self.retry_policy.clone(),
             rpc_deadlines: self.rpc_deadlines.clone(),
             execute_plan_hook: std::sync::Arc::clone(&self.execute_plan_hook),
+            closed: Default::default(),
         }
     }
 
+    /// Close this client, as PySpark's `SparkConnectClient.close` closes its 
channel: its RPCs
+    /// raise `NO_ACTIVE_SESSION` from now on, and its clones with them. 
`release_session` is
+    /// still sent, as `stop()` sends it before it closes the client.
+    pub fn close(&self) {
+        self.closed.store(true, std::sync::atomic::Ordering::SeqCst);
+    }
+
+    /// Whether the client is closed. See [`SparkConnectClient::close`].
+    pub fn is_closed(&self) -> bool {
+        self.closed.load(std::sync::atomic::Ordering::SeqCst)
+    }
+
+    /// The error of an RPC of a closed client, as PySpark's 
`SparkConnectClient._stub` raises it.
+    #[allow(clippy::result_large_err)] // SparkError is large; changing the 
type would break the API
+    fn ensure_open(&self) -> Result<()> {
+        if self.is_closed() {
+            return Err(SparkError::classed(
+                crate::error::SparkErrorKind::Connect,
+                "NO_ACTIVE_SESSION",
+                &[],
+            ));
+        }
+        Ok(())
+    }
+
     /// Set the hook of the executions of this client and of its clones, or 
clear it.
     pub fn set_execute_plan_hook(&self, hook: Option<ExecutePlanHook>) {
         *self
@@ -521,6 +551,7 @@ impl SparkConnectClient {
         &self,
         mut request: ExecutePlanRequest,
     ) -> Result<Streaming<ExecutePlanResponse>> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         self.fill_user_context(&mut request.user_context);
         let request = self.hooked(request)?;
@@ -538,6 +569,7 @@ impl SparkConnectClient {
     /// and deep plans don't hit the recursion limit - the server sees exactly 
the bytes
     /// the reference client built.
     pub async fn execute_plan_raw(&self, request: Vec<u8>) -> 
Result<Streaming<Vec<u8>>> {
+        self.ensure_open()?;
         let mut grpc = tonic::client::Grpc::new(self.channel.clone())
             .max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
             .max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -556,6 +588,7 @@ impl SparkConnectClient {
 
     /// ReattachExecute, byte-exact passthrough returning a raw response 
stream.
     pub async fn reattach_execute_raw(&self, request: Vec<u8>) -> 
Result<Streaming<Vec<u8>>> {
+        self.ensure_open()?;
         let mut grpc = tonic::client::Grpc::new(self.channel.clone())
             .max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
             .max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -579,6 +612,7 @@ impl SparkConnectClient {
         &self,
         mut request: ReattachExecuteRequest,
     ) -> Result<Streaming<ExecutePlanResponse>> {
+        self.ensure_open()?;
         self.fill_user_context(&mut request.user_context);
         let mut req = Request::new(request);
         self._attach_metadata(&mut req);
@@ -595,6 +629,7 @@ impl SparkConnectClient {
         &self,
         mut request: ReleaseExecuteRequest,
     ) -> Result<ReleaseExecuteResponse> {
+        self.ensure_open()?;
         self.fill_user_context(&mut request.user_context);
         let mut req = Request::new(request);
         self._attach_metadata(&mut req);
@@ -612,6 +647,7 @@ impl SparkConnectClient {
     /// returns the raw response bytes, avoiding the client-side limit 
(matching the
     /// reference client, which never re-decodes the request it built).
     pub async fn analyze_plan_raw(&self, request: Vec<u8>) -> Result<Vec<u8>> {
+        self.ensure_open()?;
         let mut grpc = tonic::client::Grpc::new(self.channel.clone())
             .max_decoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT)
             .max_encoding_message_size(GRPC_MAX_MESSAGE_LENGTH_DEFAULT);
@@ -638,6 +674,7 @@ impl SparkConnectClient {
         &self,
         mut request: AnalyzePlanRequest,
     ) -> Result<AnalyzePlanResponse> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
@@ -691,6 +728,7 @@ impl SparkConnectClient {
         &self,
         mut request: ExecutePlanRequest,
     ) -> Result<ReattachableResponseStream> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         // Before the iterator: its ReattachExecute and ReleaseExecute 
requests take the
         // user context of this request, and the request of the hook, as 
PySpark's iterator
@@ -748,6 +786,7 @@ impl SparkConnectClient {
         &self,
         mut request: FetchErrorDetailsRequest,
     ) -> Result<FetchErrorDetailsResponse> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         self.fill_user_context(&mut request.user_context);
         let deadline = self.rpc_deadlines.fetch_error_details;
@@ -842,6 +881,7 @@ impl SparkConnectClient {
     ///
     /// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.config`.
     pub async fn config(&self, mut request: ConfigRequest) -> 
Result<ConfigResponse> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
@@ -862,6 +902,7 @@ impl SparkConnectClient {
     ///
     /// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.interrupt`.
     pub async fn interrupt(&self, mut request: InterruptRequest) -> 
Result<InterruptResponse> {
+        self.ensure_open()?;
         self.extend_user_context(&mut request.user_context);
         self.fill_user_context(&mut request.user_context);
         self.with_retry(|| {
@@ -1167,6 +1208,7 @@ impl SparkConnectClient {
     /// and catalog state, into a new session, of the id `new_session_id` if 
one is given. The
     /// returned client, of the same channel, is of that session.
     pub async fn clone_session(&self, new_session_id: Option<&str>) -> 
Result<Self> {
+        self.ensure_open()?;
         let request = CloneSessionRequest {
             session_id: self.session_id.clone(),
             user_context: Some(UserContext {
@@ -1232,6 +1274,7 @@ impl SparkConnectClient {
         archive: bool,
         file: bool,
     ) -> Result<()> {
+        self.ensure_open()?;
         // Each path is named as PySpark's ArtifactManager names it, by the 
flags and its file
         // extension (see `parse_artifact`).
         for path in paths {
@@ -1249,6 +1292,7 @@ impl SparkConnectClient {
     /// Used by `SparkSession::copy_from_local_to_fs`, which names the artifact
     /// `forward_to_fs/<dest>` so the server writes it to the target 
filesystem path.
     pub async fn add_named_artifact(&self, name: &str, local_path: &str) -> 
Result<()> {
+        self.ensure_open()?;
         let artifact =
             NamedArtifact::new(name.to_string(), 
Box::new(FileArtifact::new(local_path)));
         self.upload_artifact(&artifact).await
@@ -1260,6 +1304,7 @@ impl SparkConnectClient {
     /// As `SparkConnectClient.cache_artifacts`, the status check and the 
upload are one
     /// attempt under the retry policy, so a transient failure of either 
retries both.
     pub async fn cache_artifacts(&self, blobs: Vec<Vec<u8>>) -> 
Result<Vec<String>> {
+        self.ensure_open()?;
         use bytes::Bytes;
         use std::collections::HashSet;
 
@@ -1383,6 +1428,7 @@ impl SparkConnectClient {
         &self,
         names: &[&str],
     ) -> Result<std::collections::HashMap<String, bool>> {
+        self.ensure_open()?;
         let names: Vec<String> = names.iter().map(|n| n.to_string()).collect();
         match self.artifact_status_rpc(&names).await {
             Ok(statuses) => Ok(statuses),
diff --git a/crates/spark-connect/src/session.rs 
b/crates/spark-connect/src/session.rs
index 9343f736..83ce8898 100644
--- a/crates/spark-connect/src/session.rs
+++ b/crates/spark-connect/src/session.rs
@@ -903,10 +903,13 @@ impl SparkSession {
     /// Copy a local file to the driver's filesystem at `dest_path`.
     ///
     /// Mirrors `pyspark.sql.SparkSession.copyFromLocalToFs`: uploads the file 
as a
-    /// `forward_to_fs/<dest_path>` artifact, which the server writes to 
`dest_path`.
+    /// `forward_to_fs<dest_path>` artifact, which the server writes to 
`dest_path`. As the
+    /// reference, `dest_path` is an absolute path, so the name is the prefix 
and the path.
     pub fn copy_from_local_to_fs(&self, local_path: &str, dest_path: &str) -> 
Result<()> {
-        let name = format!("forward_to_fs/{}", dest_path);
-        block_on(self.client.add_named_artifact(&name, local_path))
+        block_on(
+            self.client
+                .add_named_artifact(&forward_to_fs_artifact_name(dest_path), 
local_path),
+        )
     }
 
     /// Register a user-defined function on the session so it can be 
referenced by
@@ -1026,7 +1029,11 @@ impl SparkSession {
         if self.stopped.swap(true, Ordering::SeqCst) {
             return Ok(());
         }
-        block_on(self.client.release_session())
+        // As PySpark's `stop()`, the client is closed after the session is 
released, so that
+        // its requests raise `NO_ACTIVE_SESSION` from now on.
+        let released = block_on(self.client.release_session());
+        self.client.close();
+        released
     }
 
     /// Release the session on the server, as 
`SparkConnectClient.release_session`: its state
@@ -1039,6 +1046,7 @@ impl SparkSession {
     /// closes its channel: it is stopped, and `stop()` does nothing more.
     pub fn close(&self) {
         self.stopped.store(true, Ordering::SeqCst);
+        self.client.close();
     }
 
     /// Whether `stop()` has been called on this session. Mirrors
@@ -1048,6 +1056,12 @@ impl SparkSession {
     }
 }
 
+/// The name of the artifact that copies a file to `dest_path`, as the 
reference's
+/// `FORWARD_TO_FS_PREFIX + dest_path`: the prefix is followed by the absolute 
path itself.
+fn forward_to_fs_artifact_name(dest_path: &str) -> String {
+    format!("forward_to_fs{}", dest_path)
+}
+
 impl Clone for SparkSession {
     fn clone(&self) -> Self {
         SparkSession {
@@ -2618,6 +2632,28 @@ mod tests {
         assert!(spark.stop().is_ok());
     }
 
+    #[test]
+    fn requests_of_a_stopped_session_raise_no_active_session() {
+        // Port 1: nothing listens, so a request that reached the wire would 
fail differently.
+        // As PySpark's closed client, the stopped session raises before it 
sends anything.
+        let spark = SparkSession::builder()
+            .remote("sc://localhost:1")
+            .get_or_create()
+            .expect("failed to build session");
+        let _ = spark.stop();
+        let err = spark
+            .conf()
+            .get("spark.sql.shuffle.partitions")
+            .expect_err("a request of a stopped session must fail");
+        assert_eq!(err.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+    }
+
+    #[test]
+    fn forward_to_fs_artifact_name_is_the_prefix_and_the_absolute_path() {
+        // As the reference's `FORWARD_TO_FS_PREFIX + dest_path`, with no 
separator added.
+        assert_eq!(forward_to_fs_artifact_name("/tmp/x"), 
"forward_to_fs/tmp/x");
+    }
+
     #[test]
     fn version_cache_follows_session_identity() {
         let spark = session();
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index a26457d5..60563cd1 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -760,7 +760,8 @@ def test_file_upload_attribution_on_every_frame(
             destinations = ["/wire/small.txt", "/wire/large.txt"]
             for path, destination in zip([small, large], destinations):
                 spark.copyFromLocalToFs(str(path), destination)
-            artifact_names = ["forward_to_fs/" + destination for destination 
in destinations]
+            # As PySpark's FORWARD_TO_FS_PREFIX + dest_path: the path is 
absolute, so no separator.
+            artifact_names = ["forward_to_fs" + destination for destination in 
destinations]
         payloads = {context["payload"] for context in state["upload_contexts"]}
         assert payloads == {"batch", "begin_chunk", "chunk"}, (
             f"Expected all artifact frame kinds, got {payloads}. "
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 5972870e..4afd6911 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -2852,6 +2852,36 @@ def 
test_streaming_queries_return_the_status_and_the_exception_of_pyspark(sessio
         failed.stop()
 
 
+def test_builder_config_none_leaves_the_key_unset_as_pyspark(session):
+    # As PySpark's `_set_all`: an option of None, of config or of map, leaves 
its key unset on
+    # the session, while the other options of the builder are set.
+    built = (SparkSession.builder.remote(os.environ["SPARK_REMOTE"])
+             .config("spark.r12.none", None)
+             .config("spark.r12.kept", "v")
+             .config(map={"spark.r12.map": None})
+             .getOrCreate())
+    assert built.session_id == session.session_id
+    assert session.conf.get("spark.r12.kept") == "v"
+    assert session.conf.get("spark.r12.none", "unset") == "unset"
+    assert session.conf.get("spark.r12.map", "unset") == "unset"
+    assert "spark.r12.none" not in session.conf.getAll
+    with pytest.raises(Exception) as e:
+        session.conf.get("spark.r12.none")
+    assert e.value.getCondition() == "SQL_CONF_NOT_FOUND"
+
+
+def 
test_stopped_session_raises_no_active_session_while_the_others_work(session):
+    from pyspark.errors.exceptions.connect import SparkConnectException
+
+    other = session.newSession()
+    other.stop()
+    assert other.is_stopped
+    with pytest.raises(SparkConnectException) as e:
+        other.range(2).count()
+    assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+    assert session.range(3).count() == 3
+
+
 def 
test_stream_writer_starts_a_query_of_its_real_time_trigger_as_pyspark(session):
     # As PySpark's, the real-time trigger reaches the server, which runs 
real-time mode for
     # some sinks only, not the memory sink.
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 2ce2d6d4..08f61f51 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -11052,6 +11052,38 @@ def 
test_positional_indexing_integer_subclass_and_string_lookup(positional_index
     assert service.calls == before
 
 
+def test_copy_from_local_to_fs_requires_an_absolute_destination(mock_connect, 
tmp_path):
+    # As PySpark Connect's `_parse_forward_to_fs_artifacts`: the destination 
must be an absolute
+    # path, which is checked before any request, so the server never sees a 
relative one.
+    spark, _ = mock_connect
+    local = tmp_path / "local.txt"
+    local.write_text("x")
+    with pytest.raises(AssertionError, match="destination FS path must be an 
absolute path."):
+        spark.copyFromLocalToFs(str(local), "relative/out.txt")
+
+
+def test_requests_of_a_stopped_session_raise_no_active_session_as_pyspark():
+    # As PySpark's closed client: stop() releases the session and closes the 
client, so each
+    # request of the stopped session, and of its DataFrames, raises 
NO_ACTIVE_SESSION.
+    from pyspark.errors.exceptions.connect import SparkConnectException
+
+    spark = _offline_session()
+    other = spark.newSession()
+    other.stop()
+    assert other.is_stopped
+    requests = [
+        lambda: other.range(2).count(),
+        lambda: other.sql("select 1").collect(),
+        lambda: other.conf.get("spark.sql.shuffle.partitions"),
+        lambda: other.version,
+    ]
+    for request in requests:
+        with pytest.raises(SparkConnectException) as e:
+            request()
+        assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+    spark.stop()
+
+
 def 
test_positional_indexing_invalid_index_types_do_not_analyze(positional_indexing_analyzed):
     from pyspark.errors import PySparkTypeError
 
@@ -12671,13 +12703,15 @@ def 
test_version_cache_active_session_handle_shares_cache(version_cache_session)
 
 
 def 
test_version_cache_stop_before_first_lookup_keeps_rpc_behavior(version_cache_session):
+    # As PySpark's closed client, a lookup after stop() raises 
NO_ACTIVE_SESSION without a request.
     spark, service = version_cache_session
     spark.stop()
     assert service.calls == 0
     service.failure = "rpc"
-    with pytest.raises(Exception):
+    with pytest.raises(Exception) as e:
         _ = spark.version
-    assert service.calls == 1
+    assert e.value.getCondition() == "NO_ACTIVE_SESSION"
+    assert service.calls == 0
 
 
 # --------------------------------------------------------------------------- 
printSchema


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to