This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new e847573  [SPARK-61008][CONNECT][PYTHON] Get or create the sessions of 
SparkSession.builder as PySpark does
e847573 is described below

commit e8475739b7920312e9d51044e7f5486095d7352d
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 22:29:52 2026 +0900

    [SPARK-61008][CONNECT][PYTHON] Get or create the sessions of 
SparkSession.builder as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61008](https://issues.apache.org/jira/browse/SPARK-61008): 
`SparkSession.builder` now gets or creates sessions as PySpark 4.3.0's does.
    
    - `getOrCreate()` returns the active session of the thread, or else the 
default session, as the same object, and creates one only when there is 
neither. It sets the options of the builder on the session that it returns.
    - The options are set in one request whose failures are ignored, as 
`_apply_options` sends. That ignores, for example, static configurations. 
`spark.remote` and `spark.master` are not options, and `appName()` sets 
`spark.app.name`.
    - The URL is that of `remote()`, or else of `SPARK_REMOTE`. Without either, 
`PySparkRuntimeError` `CONNECT_URL_NOT_SET` is raised.
    - The first session created is the default session, and the active session 
is thread-local. `create()` makes a new session, which is the default and the 
active session only when they are not set. `stop()` unsets them, and 
`SparkSession.active()` returns the active or else the default session.
    
    The core configuration has `set_all`, of several values in one request, 
silently or not, for this.
    
    ### Why are the changes needed?
    
    `getOrCreate()` created a new session each time, so the configurations and 
temporary views of a session were not those of the next one. After 
`spark.conf.set("spark.sql.shuffle.partitions", "7")` and 
`createOrReplaceTempView("v")`, a second `getOrCreate()` returned a session 
whose configuration was `200` and that had no view `v`. A library that calls 
`getOrCreate()` got a session of its own. `getOrCreate()` also raised 
`ValueError` without `remote()`, even with `SPARK_REMOTE` set. One [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. `getOrCreate()`, `create()`, `getActiveSession()` and `active()` 
behave as in PySpark 4.3.0, and an existing session is returned with the 
options of the builder set on it.
    
    ### How was this patch tested?
    
    - A new offline test checks, in a new interpreter:
      - `getOrCreate()` returns the active session, as the same object, 
whatever the URL, and the default session in another thread.
      - `create()` does not replace the active session.
      - `stop()` unsets the sessions.
      - The URL comes from `SPARK_REMOTE`, and `CONNECT_URL_NOT_SET` and 
`NO_ACTIVE_OR_DEFAULT_SESSION` are raised.
    - A new offline test checks the options that a mock Connect server 
receives. They arrive in one silent request, without `spark.master`, and once 
for an existing session too. The mock now serves Config `Set` requests.
    - A new live test checks that `getOrCreate()` returns the session with an 
option set, and that a static configuration is ignored. It also passes with 
official PySpark 4.3.0.
    - The new tests fail without this change.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of 
all tests with `wasm-udf`, the offline Python suites (1161 passed, 8 skipped), 
and the live drop-in and Arrow suites (87 passed) and live cases (10 passed). 
Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #211 from HyukjinKwon/SPARK-61008-get-or-create.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs        | 210 ++++++++++++++++++++++++--------
 crates/spark-connect-core/src/client.rs |  31 +++++
 crates/spark-connect/src/conf.rs        |   6 +
 python/tests/_mock_connect_server.py    |  16 ++-
 python/tests/test_dropin_e2e.py         |  19 +++
 python/tests/test_dropin_offline.py     |  71 +++++++++++
 6 files changed, 303 insertions(+), 50 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 2536399..dafb3cf 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -107,13 +107,13 @@ impl PySparkSessionBuilder {
         Ok(b)
     }
 
-    /// `appName` - accepted for API parity; a Connect client cannot rename an
-    /// already-running remote server, so this is a no-op (chainable), matching
-    /// pyspark's Connect behavior.
+    /// `appName` - the option `spark.app.name`, as PySpark's builder sets. 
Like any option,
+    /// it is set on the session silently, so it has no effect on a running 
server.
     #[pyo3(name = "appName")]
     fn app_name(&self, name: &str) -> PySparkSessionBuilder {
-        let _ = name;
-        self.clone()
+        let mut b = self.clone();
+        b.set_conf("spark.app.name", name.to_string());
+        b
     }
 
     /// `master` - not applicable to a remote Connect session (no local 
cluster to
@@ -184,29 +184,36 @@ impl PySparkSessionBuilder {
         self.clone()
     }
 
-    /// Build and get or create the session (`getOrCreate`).
+    /// Get the active session of this thread, or else the default session, or 
create one
+    /// (`getOrCreate`), and set the options of the builder on it. Mirrors 
PySpark's: the URL
+    /// is that of `remote()` or else of `SPARK_REMOTE`, and an existing 
session is returned
+    /// whatever its URL.
     #[pyo3(name = "getOrCreate")]
-    fn get_or_create(&self, py: Python<'_>) -> PyResult<PySparkSession> {
-        let url = self.remote_url.clone().ok_or_else(|| {
-            pyo3::exceptions::PyValueError::new_err("Must call .remote(url) 
before .getOrCreate()")
-        })?;
-
-        let builder = SparkSession::builder().remote(&url);
-        let session = py.detach(|| builder.get_or_create()).to_pyerr()?;
-        // Apply builder configs as runtime confs now that we have a session.
-        let conf = session.conf();
-        for (k, v) in &self.configs {
-            py.detach(|| conf.set(k, v)).to_pyerr()?;
-        }
-        *active_slot().lock().unwrap() = Some(session.clone());
+    fn get_or_create(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+        let url = self.url(py)?;
+        let session = {
+            let _creating = session_lock();
+            match active_session(py).or_else(|| default_session(py)) {
+                Some(session) => session,
+                None => new_session(py, &url)?,
+            }
+        };
+        self.apply_options(py, &session.borrow(py).session)?;
         log_client_banner(py);
-        Ok(PySparkSession::new(session))
+        Ok(session)
     }
 
-    /// `create()` - builds a brand-new session. For Connect this is 
equivalent to
-    /// getOrCreate against the configured remote.
-    fn create(&self, py: Python<'_>) -> PyResult<PySparkSession> {
-        self.get_or_create(py)
+    /// `create()` - a new session, which is the default session, and the 
active one of this
+    /// thread, only when they are not set. Mirrors PySpark's.
+    fn create(&self, py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+        let url = self.url(py)?;
+        let session = {
+            let _creating = session_lock();
+            new_session(py, &url)?
+        };
+        self.apply_options(py, &session.borrow(py).session)?;
+        log_client_banner(py);
+        Ok(session)
     }
 }
 
@@ -219,15 +226,115 @@ impl PySparkSessionBuilder {
             self.configs.push((key.to_string(), value));
         }
     }
+
+    /// The URL of the session: that of `remote()`, or else of `SPARK_REMOTE`, 
as PySpark's
+    /// builder of a Connect-only client, which raises `CONNECT_URL_NOT_SET` 
without either.
+    fn url(&self, py: Python<'_>) -> PyResult<String> {
+        match self
+            .remote_url
+            .clone()
+            .or_else(|| std::env::var("SPARK_REMOTE").ok())
+        {
+            Some(url) => Ok(url),
+            None => Err(crate::column::raise_pyspark(
+                py,
+                "PySparkRuntimeError",
+                "CONNECT_URL_NOT_SET",
+                &[],
+            )),
+        }
+    }
+
+    /// Set the options of the builder on `session`, as PySpark's 
`_apply_options`: in one
+    /// request whose failures are ignored, such as those of static 
configurations, and
+    /// without `spark.master`, which a session does not take.
+    // SparkError is large, as the core crate allows for its API.
+    #[allow(clippy::result_large_err)]
+    fn apply_options(&self, py: Python<'_>, session: &SparkSession) -> 
PyResult<()> {
+        let options: Vec<(String, String)> = self
+            .configs
+            .iter()
+            .filter(|(key, _)| key != "spark.master")
+            .cloned()
+            .collect();
+        if options.is_empty() {
+            return Ok(());
+        }
+        let conf = session.conf();
+        py.detach(|| conf.set_all(&options, true)).to_pyerr()
+    }
 }
 
-/// Process-global "active" session (mirrors 
`SparkSession.active`/`getActiveSession`).
-fn active_slot() -> &'static std::sync::Mutex<Option<SparkSession>> {
-    static SLOT: std::sync::OnceLock<std::sync::Mutex<Option<SparkSession>>> =
+/// The default session: the first session created, of every thread, until it 
stops. Mirrors
+/// `SparkSession._default_session`. The Python object, which `getOrCreate()` 
returns as it is.
+fn default_slot() -> &'static std::sync::Mutex<Option<Py<PySparkSession>>> {
+    static SLOT: 
std::sync::OnceLock<std::sync::Mutex<Option<Py<PySparkSession>>>> =
         std::sync::OnceLock::new();
     SLOT.get_or_init(|| std::sync::Mutex::new(None))
 }
 
+thread_local! {
+    /// The active session of this thread, until it stops. Mirrors the 
thread-local
+    /// `SparkSession._active_session`.
+    static ACTIVE_SESSION: std::cell::RefCell<Option<Py<PySparkSession>>> =
+        const { std::cell::RefCell::new(None) };
+}
+
+/// Held while a session is got or created, as PySpark's `SparkSession._lock`, 
so that threads
+/// get the same default session. Taken with the GIL, which is not released 
while it is held.
+fn session_lock() -> std::sync::MutexGuard<'static, ()> {
+    static LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+    LOCK.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// `session`, unless it stopped.
+fn unless_stopped(py: Python<'_>, session: &Py<PySparkSession>) -> 
Option<Py<PySparkSession>> {
+    (!session.borrow(py).session.is_stopped()).then(|| session.clone_ref(py))
+}
+
+/// The active session of this thread, unless it stopped.
+fn active_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+    ACTIVE_SESSION.with(|active| {
+        active
+            .borrow()
+            .as_ref()
+            .and_then(|session| unless_stopped(py, session))
+    })
+}
+
+/// The default session, unless it stopped.
+fn default_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+    default_slot()
+        .lock()
+        .unwrap_or_else(|e| e.into_inner())
+        .as_ref()
+        .and_then(|session| unless_stopped(py, session))
+}
+
+/// A new session of `url`, which is the default session, and the active 
session of this
+/// thread, when they are not set, as PySpark's 
`_set_default_and_active_session`.
+fn new_session(py: Python<'_>, url: &str) -> PyResult<Py<PySparkSession>> {
+    // The channel connects lazily, so this does not wait on the server.
+    let session = SparkSession::builder()
+        .remote(url)
+        .get_or_create()
+        .to_pyerr()?;
+    let session = Py::new(py, PySparkSession::new(session))?;
+    {
+        let mut default = default_slot().lock().unwrap_or_else(|e| 
e.into_inner());
+        if default.is_none() {
+            *default = Some(session.clone_ref(py));
+        }
+    }
+    ACTIVE_SESSION.with(|active| {
+        let mut active = active.borrow_mut();
+        if active.is_none() {
+            *active = Some(session.clone_ref(py));
+        }
+    });
+    Ok(session)
+}
+
 /// Emit a one-time INFO log identifying this as the Rust-backed Spark Connect
 /// client, so it is discoverable at runtime which client is in use -- and 
issues
 /// get filed against this project, not the reference `pyspark-client`. Logged 
the
@@ -996,13 +1103,22 @@ impl PySparkSession {
                 ),),
             )?;
         }
+        // As PySpark's, a stopped session is no longer the default session, 
or the active
+        // session of this thread.
         let sid = self.session.session_id();
+        let is_this = |s: &Py<PySparkSession>| 
s.borrow(py).session.session_id() == sid;
         {
-            let mut slot = active_slot().lock().unwrap();
-            if slot.as_ref().is_some_and(|s| s.session_id() == sid) {
-                *slot = None;
+            let mut default = default_slot().lock().unwrap_or_else(|e| 
e.into_inner());
+            if default.as_ref().is_some_and(is_this) {
+                *default = None;
             }
         }
+        ACTIVE_SESSION.with(|active| {
+            let mut active = active.borrow_mut();
+            if active.as_ref().is_some_and(is_this) {
+                *active = None;
+            }
+        });
         // Only touch the SQLContext cache if the module was ever imported (as 
pyspark does).
         let modules = py.import("sys")?.getattr("modules")?;
         if let Ok(ctx_mod) = modules.get_item("pyspark.sql.connect.context") {
@@ -1139,29 +1255,27 @@ impl PySparkSession {
             .call0()
     }
 
-    /// The active session for this process, or None. Mirrors 
`SparkSession.getActiveSession`.
+    /// The active session of this thread, or None. Mirrors 
`SparkSession.getActiveSession`.
     #[staticmethod]
     #[pyo3(name = "getActiveSession")]
-    fn get_active_session() -> Option<PySparkSession> {
-        active_slot()
-            .lock()
-            .unwrap()
-            .clone()
-            .map(PySparkSession::new)
+    fn get_active_session(py: Python<'_>) -> Option<Py<PySparkSession>> {
+        active_session(py)
     }
 
-    /// The active session; errors if none. Mirrors `SparkSession.active`, 
which raises
-    /// `NO_ACTIVE_OR_DEFAULT_SESSION`.
+    /// The active session of this thread, or else the default session; errors 
if neither.
+    /// Mirrors `SparkSession.active`, which raises 
`NO_ACTIVE_OR_DEFAULT_SESSION`.
     #[staticmethod]
-    fn active(py: Python<'_>) -> PyResult<PySparkSession> {
-        Self::get_active_session().ok_or_else(|| {
-            crate::column::raise_pyspark(
-                py,
-                "PySparkRuntimeError",
-                "NO_ACTIVE_OR_DEFAULT_SESSION",
-                &[],
-            )
-        })
+    fn active(py: Python<'_>) -> PyResult<Py<PySparkSession>> {
+        active_session(py)
+            .or_else(|| default_session(py))
+            .ok_or_else(|| {
+                crate::column::raise_pyspark(
+                    py,
+                    "PySparkRuntimeError",
+                    "NO_ACTIVE_OR_DEFAULT_SESSION",
+                    &[],
+                )
+            })
     }
 
     /// Register a progress handler invoked on each execution-progress message.
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 1c1881a..09ca9c4 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -701,6 +701,37 @@ impl SparkConnectClient {
         Ok(())
     }
 
+    /// Set configuration values in one request, ignoring the failures to set 
them when
+    /// `silent`. Mirrors `pyspark.sql.connect.conf.RuntimeConf._set_all`, 
with which the
+    /// session builder applies its options.
+    pub async fn set_configs(&self, pairs: &[(String, String)], silent: bool) 
-> Result<()> {
+        let set = spark_connect_proto::config_request::Set {
+            pairs: pairs
+                .iter()
+                .map(|(key, value)| spark_connect_proto::KeyValue {
+                    key: key.clone(),
+                    value: Some(value.clone()),
+                })
+                .collect(),
+            silent: Some(silent),
+        };
+        let request = ConfigRequest {
+            session_id: self.session_id.clone(),
+            user_context: Some(UserContext {
+                user_id: self.user_id.clone().unwrap_or_default(),
+                ..Default::default()
+            }),
+            operation: Some(spark_connect_proto::config_request::Operation {
+                op_type: 
Some(spark_connect_proto::config_request::operation::OpType::Set(
+                    set,
+                )),
+            }),
+            ..Default::default()
+        };
+        let _ = self.config(request).await?;
+        Ok(())
+    }
+
     /// Get configuration values with default fallbacks.
     ///
     /// Mirrors 
`pyspark.sql.connect.client.core.SparkConnectClient.get_config_with_defaults`.
diff --git a/crates/spark-connect/src/conf.rs b/crates/spark-connect/src/conf.rs
index 3227f11..10d38c1 100644
--- a/crates/spark-connect/src/conf.rs
+++ b/crates/spark-connect/src/conf.rs
@@ -43,6 +43,12 @@ impl RuntimeConf {
         block_on(self.client.set_config(key, value))
     }
 
+    /// Set several configuration values in one request, ignoring the failures 
to set them,
+    /// such as of static configurations, when `silent`. Mirrors 
`RuntimeConf._set_all`.
+    pub fn set_all(&self, pairs: &[(String, String)], silent: bool) -> 
Result<()> {
+        block_on(self.client.set_configs(pairs, silent))
+    }
+
     /// Get a configuration value by key.
     ///
     /// Returns `Ok(None)` if the key does not exist on the server.
diff --git a/python/tests/_mock_connect_server.py 
b/python/tests/_mock_connect_server.py
index c2ea1eb..53a5e6c 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -125,6 +125,7 @@ def serve(connection):
             self.fetched_error_ids = []
             self.confs = {}
             self.read_confs = []
+            self.set_confs = []
 
         def analyze(self, request, context):
             self.calls += 1
@@ -160,7 +161,8 @@ def serve(connection):
         # FetchErrorDetails returns `error_details` (raw 
FetchErrorDetailsResponse fields) and
         # records the error id asked for. Config Get returns the values in 
`confs`, or else
         # those of `default_confs`, the defaults of SQL configs of the server, 
and fails for a
-        # key without one, as the server does for an unset config; it records 
the keys.
+        # key without one, as the server does for an unset config; it records 
the keys. Config
+        # Set records the pairs and the silent flag of each request.
         error_details = b""
         fetched_error_ids = []
         confs = {}
@@ -169,6 +171,7 @@ def serve(connection):
             "spark.sql.execution.pythonUDTF.arrow.enabled": "true",
         }
         read_confs = []
+        set_confs = []
 
         def execute(self, request, context):
             self.execute_calls += 1
@@ -220,8 +223,17 @@ def serve(connection):
             # ConfigResponse.session_id=1, pairs=2, KeyValue.key=1, value=2.
             request = parse_message(request)
             operation = parse_message(request[3][0])
+            if 1 in operation:
+                # Operation.set=1: Set.pairs=1 (KeyValue.key=1, value=2), 
silent=2.
+                set_request = parse_message(operation[1][0])
+                pairs = []
+                for pair in set_request.get(1, []):
+                    pair = parse_message(pair)
+                    pairs.append((pair[1][0].decode(), pair[2][0].decode()))
+                self.set_confs.append((pairs, set_request.get(2, [None])[0]))
+                return message_field(1, request[1][0])
             if 2 not in operation:
-                context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get is 
served")
+                context.abort(grpc.StatusCode.UNIMPLEMENTED, "only Get and Set 
are served")
             keys = [key.decode() for key in 
parse_message(operation[2][0]).get(1, [])]
             self.read_confs.extend(keys)
             confs = {**self.default_confs, **self.confs}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index e604d25..1472a4b 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -74,6 +74,25 @@ def test_conf_get_with_a_none_default(session):
         session.conf.get(key)
 
 
+def 
test_get_or_create_returns_the_session_with_the_options_as_pyspark(session):
+    # As PySpark's builder: getOrCreate() returns the active session, and sets 
the options
+    # of the builder on it; one that fails, of a static configuration, is 
ignored.
+    key = "spark.sql.shuffle.partitions"
+    previous = session.conf.get(key)
+    try:
+        got = (
+            SparkSession.builder.remote(os.environ["SPARK_REMOTE"])
+            .config(key, "5")
+            .config("spark.sql.warehouse.dir", "/not/set")
+            .getOrCreate()
+        )
+        assert got is session
+        assert session.conf.get(key) == "5"
+        assert session.conf.get("spark.sql.warehouse.dir") != "/not/set"
+    finally:
+        session.conf.set(key, previous)
+
+
 def test_repr_shows_the_rows_with_eager_evaluation_as_pyspark(session):
     df = session.range(3)
     assert repr(df) == "DataFrame[id: bigint]"
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 65fb374..32b52f7 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -378,6 +378,77 @@ def test_session_builder_chaining_offline():
     assert type(b2).__name__ == "SparkSessionBuilder"
 
 
+def test_sessions_are_got_or_created_as_pyspark():
+    # As PySpark's builder: getOrCreate() returns the active session of the 
thread, or else
+    # the default session, whatever the URL, and creates one when there is 
neither, of the
+    # URL of remote() or else of SPARK_REMOTE; create() makes a new one, which 
is the default
+    # and the active session only when they are not set; and stop() unsets 
them. In a new
+    # interpreter, which has no default session.
+    import subprocess
+    import sys
+
+    subprocess.run([sys.executable, "-c", """
+import os
+import threading
+
+from pyspark.errors import PySparkRuntimeError
+from pyspark.sql import SparkSession
+
+def condition(call):
+    try:
+        call()
+    except PySparkRuntimeError as error:
+        return error.getCondition()
+    raise AssertionError("no error")
+
+os.environ.pop("SPARK_REMOTE", None)
+assert condition(SparkSession.builder.getOrCreate) == "CONNECT_URL_NOT_SET"
+assert condition(SparkSession.active) == "NO_ACTIVE_OR_DEFAULT_SESSION"
+
+first = SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+assert SparkSession.builder.remote("sc://localhost:2").getOrCreate() is first
+assert SparkSession.getActiveSession() is first and SparkSession.active() is 
first
+
+seen = {}
+def other_thread():
+    seen["active"] = SparkSession.getActiveSession()
+    seen["default"] = SparkSession.active()
+    seen["got"] = SparkSession.builder.remote("sc://localhost:1").getOrCreate()
+thread = threading.Thread(target=other_thread)
+thread.start()
+thread.join()
+assert seen == {"active": None, "default": first, "got": first}, seen
+
+created = SparkSession.builder.remote("sc://localhost:1").create()
+assert created is not first and SparkSession.getActiveSession() is first
+
+first.stop()
+assert SparkSession.getActiveSession() is None
+assert condition(SparkSession.active) == "NO_ACTIVE_OR_DEFAULT_SESSION"
+os.environ["SPARK_REMOTE"] = "sc://localhost:1"
+again = SparkSession.builder.getOrCreate()
+assert again is not first and again is not created
+assert SparkSession.getActiveSession() is again and SparkSession.active() is 
again
+"""], check=True)
+
+
+def test_builder_options_are_set_on_the_session_as_pyspark(mock_connect):
+    # As PySpark's _apply_options: the options of the builder, but 
spark.remote and
+    # spark.master, are set on the session that getOrCreate() returns, also an 
existing one,
+    # in one request whose failures are ignored.
+    from pyspark.sql import SparkSession
+
+    spark, service = mock_connect
+    builder = 
SparkSession.builder.remote("sc://localhost:1").master("local[4]")
+    builder = builder.appName("app").config("spark.sql.shuffle.partitions", 3)
+    assert builder.getOrCreate() is spark
+    assert service.set_confs == [
+        ([("spark.app.name", "app"), ("spark.sql.shuffle.partitions", "3")], 1)
+    ]
+    assert SparkSession.builder.remote("sc://localhost:1").getOrCreate() is 
spark
+    assert len(service.set_confs) == 1
+
+
 def test_session_builder_class_constructor():
     from pyspark.sql import SparkSession
     b = SparkSession.Builder()


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to