This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new b4c7929  [SPARK-60958][CONNECT][PYTHON] Accept url in 
SparkSession.builder.remote() as PySpark does
b4c7929 is described below

commit b4c7929f20ad6ea59efa5800af68d29a4ba7a6d6
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Fri Nov 6 12:27:10 2026 +0900

    [SPARK-60958][CONNECT][PYTHON] Accept url in SparkSession.builder.remote() 
as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-60958](https://issues.apache.org/jira/browse/SPARK-60958): 
`SparkSession.builder.remote()` now takes the URL as `url`, as PySpark's 
`SparkSession.Builder.remote(url)` does, so 
`remote(url="sc://localhost:15002")` works.
    
    The client's one builder also serves 
`pyspark.sql.connect.session.SparkSession.builder`, whose `remote` in PySpark 
is `remote(location="sc://localhost")`. So the builder still takes the URL as 
the keyword `location`, and without a URL it still uses that default. Giving 
both `url` and `location` raises `TypeError`.
    
    ### Why are the changes needed?
    
    The parameter was named `location`, so 
`SparkSession.builder.remote(url=...)` raised `TypeError: 
SparkSessionBuilder.remote() got an unexpected keyword argument 'url'`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, `SparkSession.builder.remote(url=...)` works as in PySpark 4.3.0.
    
    ### How was this patch tested?
    
    - A new offline test checks the parameters and the error for both keywords. 
It also starts a mock Connect server and checks that a session built with 
`remote(url=...)`, and one built with `remote(location=...)`, each send their 
requests to it. That part runs in a new interpreter, as the session becomes the 
active one. The test fails without this change.
    
      It checks both keywords on the one builder, so it does not run as is with 
official PySpark, whose two builders each take one of them.
    - The lib tests and every integration test, e2e included, against a local 
Spark 4.3.0 server (643 passed), the build of all tests with `wasm-udf`, the 
offline Python suites (614 passed, 8 live cases skipped), and the live drop-in 
and Arrow suites (57 passed) and live cases (10 passed). Clippy reports no 
diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #185 from HyukjinKwon/SPARK-60958-remote-url.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/pyspark-rs/src/session.rs    | 20 ++++++++++++++-----
 python/tests/test_dropin_offline.py | 39 +++++++++++++++++++++++++++++++++++++
 2 files changed, 54 insertions(+), 5 deletions(-)

diff --git a/crates/pyspark-rs/src/session.rs b/crates/pyspark-rs/src/session.rs
index 9fb6011..7a4d00c 100644
--- a/crates/pyspark-rs/src/session.rs
+++ b/crates/pyspark-rs/src/session.rs
@@ -50,12 +50,22 @@ impl PySparkSessionBuilder {
     }
 
     /// Set the remote Spark Connect server URL. Returns the builder 
(chainable).
-    /// Mirrors `SparkSession.Builder.remote(location="sc://localhost")`.
-    #[pyo3(signature = (location="sc://localhost"))]
-    fn remote(&self, location: &str) -> PySparkSessionBuilder {
+    /// Mirrors `SparkSession.Builder.remote(url)`, and the Connect session 
builder's
+    /// `remote(location="sc://localhost")`: the URL is `url` or the keyword 
`location`.
+    #[pyo3(signature = (url=None, *, location=None))]
+    fn remote(&self, url: Option<&str>, location: Option<&str>) -> 
PyResult<PySparkSessionBuilder> {
+        let url = match (url, location) {
+            (Some(_), Some(_)) => {
+                return Err(pyo3::exceptions::PyTypeError::new_err(
+                    "remote() got multiple values for the URL: give url or 
location",
+                ))
+            }
+            (Some(url), None) | (None, Some(url)) => url,
+            (None, None) => "sc://localhost",
+        };
         let mut b = self.clone();
-        b.remote_url = Some(location.to_string());
-        b
+        b.remote_url = Some(url.to_string());
+        Ok(b)
     }
 
     /// Set a config option, or several via `map`. Mirrors
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index 4fce78c..97f16fe 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -2342,6 +2342,45 @@ def test_session_builder_remote_url():
     assert type(builder2).__name__ == "SparkSessionBuilder"
 
 
+def test_session_builder_remote_takes_the_url_as_pyspark():
+    import inspect
+    import os
+    import subprocess
+    import sys
+
+    from pyspark.sql import SparkSession
+
+    # PySpark's SparkSession.Builder.remote(url), and the Connect builder's 
location.
+    assert list(inspect.signature(SparkSession.builder.remote).parameters) == 
["url", "location"]
+    with pytest.raises(TypeError):
+        SparkSession.builder.remote("sc://a:1", location="sc://b:2")
+    # Each keyword's URL is the server the session uses, in a new interpreter, 
as the
+    # session becomes the active one.
+    tests = os.path.dirname(os.path.abspath(__file__))
+    env = dict(os.environ)
+    env["PYTHONPATH"] = os.pathsep.join(filter(None, [tests, 
env.get("PYTHONPATH")]))
+    subprocess.run([sys.executable, "-c", """
+import multiprocessing
+from _mock_connect_server import RemoteService, serve
+from pyspark.sql import SparkSession
+
+parent, child = multiprocessing.get_context("spawn").Pipe()
+process = multiprocessing.get_context("spawn").Process(target=serve, 
args=(child,))
+process.start()
+try:
+    assert parent.poll(20), "Mock server did not start"
+    url = f"sc://127.0.0.1:{parent.recv()}"
+    for calls, keyword in enumerate(["url", "location"], start=1):
+        spark = SparkSession.builder.remote(**{keyword: url}).getOrCreate()
+        assert spark.range(3).columns == ["id"]
+        assert RemoteService(parent).calls == calls, keyword
+        spark.stop()
+finally:
+    parent.send(("stop",))
+    process.join(10)
+"""], check=True, env=env)
+
+
 def test_functions_struct_creation():
     """Test struct function for creating struct columns."""
     from pyspark.sql import functions as F


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to