This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new cf0d2daf [SPARK-61182][CONNECT] Connect over TLS when the connection 
string has a token for a host other than localhost, take any UUID as the 
session ID, and default the user ID to SPARK_USER or USER, as PySpark does
cf0d2daf is described below

commit cf0d2dafcf94e008e823f8a082ca1c6e17ca4f2e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 20:29:44 2026 +0900

    [SPARK-61182][CONNECT] Connect over TLS when the connection string has a 
token for a host other than localhost, take any UUID as the session ID, and 
default the user ID to SPARK_USER or USER, as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61182](https://issues.apache.org/jira/browse/SPARK-61182): PySpark 
connects over TLS when `use_ssl` is set or a token is present, a token on 
`localhost` stays on local channel credentials, and a session ID is any UUID. 
The user ID is the connection string's `user_id`, else `SPARK_USER`, else 
`USER`.
    
    - `ChannelBuilder::secure()` is the scheme predicate, and 
`SparkConnectClient::connect` dials TLS when it holds. It is true for 
`use_ssl=true`, and for a token on a host that is not loopback. A token on 
`localhost`, `127.0.0.1` or `[::1]` stays plaintext, as before. The refusal 
"Refusing to send the authentication token ... over an insecure connection" is 
removed, since the token now always goes over TLS unless the host is loopback.
    - `ChannelBuilder::session_id()` accepts any UUID that parses, and returns 
the string as given. Malformed strings still raise `INVALID_SESSION_UUID_ID`.
    - `SparkConnectClient::connect` takes the user ID from `SPARK_USER`, else 
`USER`, when the connection string sets no `user_id`. An empty value is no user 
ID. The user ID is then set in the user context of each request, as before for 
an explicit `user_id`.
    - `docs/configuration.md` says that a token implies TLS, and that a token 
is sent without TLS only to `localhost`, `127.0.0.1` and `[::1]`.
    
    ### Why are the changes needed?
    
    - A token for a host other than localhost, without `use_ssl=true`, failed 
to connect. A call such as `sc://example.com:443/;token=T`, or a token from 
`SPARK_CONNECT_AUTHENTICATE_TOKEN`, returned the refusal. PySpark connects over 
TLS for it. `DefaultChannelBuilder(...).secure` in PySpark 4.3.0 returns `True` 
for `sc://example.com/;token=T`, and for `sc://localhost/;token=T`.
    - A UUID of version 1, such as 
`sc://localhost/;session_id=6ba7b810-9dad-11d1-80b4-00c04fd430c8`, raised 
`INVALID_SESSION_UUID_ID`, since the check required version 4. PySpark accepts 
it: `DefaultChannelBuilder(...).session_id` returns the string.
    - Without `user_id` in the connection string, the user context of every 
request had an empty `user_id`. PySpark sends the OS user. With 
`SPARK_USER=hook-user`, `SparkConnectClient(...)._user_id` is `'hook-user'` in 
PySpark 4.3.0, and with only `USER=hook-os-user` it is `'hook-os-user'`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes.
    
    - A token for a host other than localhost is sent over TLS, and no longer 
refused without `use_ssl=true`.
    - A session ID of any UUID version is accepted, and sent as given.
    - Requests carry the user ID from `SPARK_USER`, else `USER`, when the 
connection string has no `user_id`. The same user ID is also sent as the 
`user_id` gRPC header, which the client already sent for an explicit `user_id`.
    - Known difference from PySpark: a token on `127.0.0.1` or `[::1]` stays 
plaintext, where PySpark uses TLS for those hosts. This is the existing 
loopback leniency, unchanged.
    
    ### How was this patch tested?
    
    - Rust unit tests in `crates/spark-connect-core/src/channel.rs`: 
`scheme_is_tls_for_a_token_on_a_remote_host`, 
`scheme_uses_the_token_of_the_environment`, 
`session_id_accepts_a_uuid_of_any_version_as_given`, 
`session_id_rejects_a_malformed_uuid`.
    - Rust unit tests in `crates/spark-connect-core/src/client.rs`: 
`test_token_without_ssl_to_remote_host_is_sent_over_tls` (replaces the refusal 
test), `test_token_of_the_environment_to_remote_host_is_sent_over_tls`, 
`test_user_id_from_the_environment_as_pyspark`, 
`test_user_id_of_the_connection_then_the_environment`.
    - `crates/spark-connect-core/tests/user_context.rs`: 
`typed_requests_keep_their_user_context_with_an_empty_user_id` now uses 
`user_id=` in the connection string, since the default would otherwise read the 
`USER` of the machine.
    - `test_cached_arrow_upload_attribution_on_every_frame` and 
`test_file_upload_attribution_on_every_frame` in 
`python/tests/test_arrow_handoff.py` set `SPARK_USER` for a connection string 
without `user_id`, and expect it as the user ID of every frame, where they 
expected an empty one.
    - Environment-variable tests take a shared lock (`channel::set_env`), so 
they do not race other tests.
    - Python offline test 
`test_session_id_of_any_uuid_version_and_the_user_id_of_the_requests` in 
`python/tests/test_dropin_offline.py`: a version 1 session ID is kept as given, 
and the `ExecutePlanRequest` seen by the mock server has that session ID and 
the user ID from `SPARK_USER`, then from `USER`.
    - The same cases checked against official PySpark 4.3.0 with 
`DefaultChannelBuilder` and `SparkConnectClient`: the session ID is accepted, 
`secure` is as above, and the user ID is `SPARK_USER`, else `USER`.
    - The Python test fails without this change, with `INVALID_SESSION_UUID_ID`.
    - The lib tests of both crates (604 passed), the build of all tests, also 
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the 
live drop-in and Arrow suites (209 passed) and live cases (10 passed) against a 
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #373 from HyukjinKwon/SPARK-61182-channel-builder-parity.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/channel.rs        | 132 ++++++++++++++++++----
 crates/spark-connect-core/src/client.rs         | 140 ++++++++++++++++++------
 crates/spark-connect-core/tests/user_context.rs |   9 +-
 docs/configuration.md                           |   6 +-
 python/tests/test_arrow_handoff.py              |   8 +-
 python/tests/test_dropin_offline.py             |  23 ++++
 6 files changed, 255 insertions(+), 63 deletions(-)

diff --git a/crates/spark-connect-core/src/channel.rs 
b/crates/spark-connect-core/src/channel.rs
index 43a9e28b..753fbba3 100644
--- a/crates/spark-connect-core/src/channel.rs
+++ b/crates/spark-connect-core/src/channel.rs
@@ -154,14 +154,15 @@ impl ChannelBuilder {
             .or_else(|| std::env::var("SPARK_CONNECT_AUTHENTICATE_TOKEN").ok())
     }
 
-    /// Mirrors the reference client's `secure` property (`use_ssl` OR a token 
is
-    /// present). NOTE: this does NOT gate the transport. It reads like a 
security
-    /// check but grants none: the token half was the source of the 
cleartext-token
-    /// bug, since a token alone reports "secure" while the scheme stays 
plaintext.
-    /// The real gate on sending the token is `use_ssl() && !is_loopback()` in
-    /// `SparkConnectClient::connect`; this method is kept only for 
parity/inspection.
+    /// Whether the channel uses TLS, which is the scheme 
`SparkConnectClient::connect` dials.
+    ///
+    /// The reference client (`core.py` `ChannelBuilder.secure`) is TLS when 
`use_ssl` is set
+    /// or a token is present. A token on a loopback host stays plaintext 
here, as a
+    /// documented leniency for local development; the reference client uses 
local channel
+    /// credentials for `localhost` in that case. A token is therefore never 
sent in
+    /// cleartext to a remote host.
     pub fn secure(&self) -> bool {
-        self.use_ssl() || self.token().is_some()
+        self.use_ssl() || (self.token().is_some() && !self.is_loopback())
     }
 
     /// Whether the endpoint is a loopback address.
@@ -169,7 +170,7 @@ impl ChannelBuilder {
     /// The reference client (`core.py` `toChannel`) permits a token over an
     /// unencrypted channel only for `host == "localhost"` (via
     /// `grpc.local_channel_credentials()`); every other host is forced onto 
TLS.
-    /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost` 
so
+    /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost`, 
so
     /// local development keeps working without `use_ssl=true`.
     pub fn is_loopback(&self) -> bool {
         matches!(self.host.as_str(), "localhost" | "127.0.0.1" | "[::1]")
@@ -179,13 +180,14 @@ impl ChannelBuilder {
         self.get(PARAM_USER_ID)
     }
 
-    /// Validates the `session_id` param is a v4 UUID (mirrors the Python 
check).
+    /// Validates the `session_id` param is a UUID of any version, and returns 
it as given.
+    /// As `uuid.UUID(session_id, version=4)` in PySpark, which does not check 
the version.
     pub fn session_id(&self) -> Result<Option<String>> {
         match self.params.get(PARAM_SESSION_ID) {
             None => Ok(None),
             Some(s) => match uuid::Uuid::parse_str(s) {
-                Ok(u) if u.get_version_num() == 4 => Ok(Some(s.clone())),
-                Ok(_) | Err(_) => Err(SparkError::value(
+                Ok(_) => Ok(Some(s.clone())),
+                Err(_) => Err(SparkError::value(
                     "INVALID_SESSION_UUID_ID",
                     &[("arg_name", "session_id"), ("origin", "invalid UUID")],
                 )),
@@ -360,12 +362,52 @@ fn quote_len(s: &str) -> usize {
     s.bytes().map(|b| if unreserved(b) { 1 } else { 3 }).sum()
 }
 
+/// Serializes the tests that set or read the environment variables that the 
channel reads,
+/// since the test threads share one environment.
+#[cfg(test)]
+pub(crate) static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+/// Sets environment variables for one test, under `ENV_LOCK`, and restores 
them on drop.
+/// A `None` value removes the variable.
+#[cfg(test)]
+pub(crate) struct EnvGuard {
+    _lock: std::sync::MutexGuard<'static, ()>,
+    saved: Vec<(&'static str, Option<std::ffi::OsString>)>,
+}
+
+#[cfg(test)]
+pub(crate) fn set_env(vars: &[(&'static str, Option<&str>)]) -> EnvGuard {
+    let lock = ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner());
+    let mut saved = Vec::new();
+    for (key, value) in vars {
+        saved.push((*key, std::env::var_os(key)));
+        match value {
+            Some(v) => std::env::set_var(key, v),
+            None => std::env::remove_var(key),
+        }
+    }
+    EnvGuard { _lock: lock, saved }
+}
+
+#[cfg(test)]
+impl Drop for EnvGuard {
+    fn drop(&mut self) {
+        for (key, value) in self.saved.iter().rev() {
+            match value {
+                Some(v) => std::env::set_var(key, v),
+                None => std::env::remove_var(key),
+            }
+        }
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
     fn basic_host_default_port() {
+        let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
         let c = ChannelBuilder::parse("sc://localhost").unwrap();
         assert_eq!(c.endpoint(), "localhost:15002");
         assert!(!c.secure());
@@ -396,10 +438,40 @@ mod tests {
     }
 
     #[test]
-    fn token_implies_secure() {
-        let c = ChannelBuilder::parse("sc://localhost/;token=xyz").unwrap();
-        assert!(c.secure());
-        assert!(!c.use_ssl());
+    fn scheme_is_tls_for_a_token_on_a_remote_host() {
+        let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+        // (url, TLS). A token goes over TLS to any host but a loopback one.
+        for (url, tls) in [
+            ("sc://example.com", false),
+            ("sc://example.com/;use_ssl=true", true),
+            ("sc://example.com/;token=T", true),
+            ("sc://example.com/;use_ssl=false;token=T", true),
+            ("sc://10.0.0.1:443/;token=T", true),
+            ("sc://localhost/;token=T", false),
+            ("sc://localhost/;use_ssl=true", true),
+            ("sc://localhost/;use_ssl=true;token=T", true),
+            ("sc://127.0.0.1/;token=T", false),
+            ("sc://127.0.0.1/;use_ssl=true", true),
+            ("sc://[::1]:15002/;token=T", false),
+            ("sc://[::1]:15002/;use_ssl=true", true),
+        ] {
+            let c = ChannelBuilder::parse(url).unwrap();
+            assert_eq!(c.secure(), tls, "{url}");
+        }
+    }
+
+    #[test]
+    fn scheme_uses_the_token_of_the_environment() {
+        let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", 
Some("ENV"))]);
+        let remote = ChannelBuilder::parse("sc://example.com").unwrap();
+        assert_eq!(remote.token().as_deref(), Some("ENV"));
+        assert!(remote.secure());
+        let local = ChannelBuilder::parse("sc://localhost").unwrap();
+        assert_eq!(local.token().as_deref(), Some("ENV"));
+        assert!(!local.secure());
+        // The token of the connection string wins over the environment.
+        let explicit = 
ChannelBuilder::parse("sc://example.com/;token=T").unwrap();
+        assert_eq!(explicit.token().as_deref(), Some("T"));
     }
 
     #[test]
@@ -447,13 +519,27 @@ mod tests {
     }
 
     #[test]
-    fn session_id_must_be_uuid4() {
-        let c = ChannelBuilder::parse(
-            "sc://localhost/;session_id=550e8400-e29b-41d4-a716-446655440000",
-        )
-        .unwrap();
-        assert!(c.session_id().is_ok());
-        let bad = 
ChannelBuilder::parse("sc://localhost/;session_id=not-a-uuid").unwrap();
-        assert!(bad.session_id().is_err());
+    fn session_id_accepts_a_uuid_of_any_version_as_given() {
+        // Versions 1, 4, 7 and the nil UUID, as `uuid.UUID(session_id, 
version=4)` accepts.
+        for id in [
+            "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
+            "550e8400-e29b-41d4-a716-446655440000",
+            "018f0b8e-0000-7000-8000-000000000000",
+            "00000000-0000-0000-0000-000000000000",
+        ] {
+            let c = 
ChannelBuilder::parse(&format!("sc://localhost/;session_id={id}")).unwrap();
+            assert_eq!(c.session_id().unwrap().as_deref(), Some(id));
+        }
+        let none = ChannelBuilder::parse("sc://localhost").unwrap();
+        assert_eq!(none.session_id().unwrap(), None);
+    }
+
+    #[test]
+    fn session_id_rejects_a_malformed_uuid() {
+        for id in ["not-a-uuid", "550e8400-e29b-41d4-a716-44665544000", "x"] {
+            let c = 
ChannelBuilder::parse(&format!("sc://localhost/;session_id={id}")).unwrap();
+            let e = c.session_id().unwrap_err();
+            assert_eq!(e.error_class, "INVALID_SESSION_UUID_ID", "{id}");
+        }
     }
 }
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index aedbec9e..7d940384 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -293,7 +293,7 @@ impl SparkConnectClient {
         // client can be constructed without a live server (e.g. in unit tests 
that only
         // build request protos).
         let endpoint = builder.endpoint();
-        let channel = if builder.use_ssl() {
+        let channel = if builder.secure() {
             // TLS connection with native root certificates
             // `with_native_roots()` is required: tonic >=0.11 does NOT load a 
trust store
             // from the `tls-native-roots` feature alone, so without this the 
TLS handshake
@@ -330,24 +330,9 @@ impl SparkConnectClient {
         // excluded from `metadata()`, so attach it explicitly here; this then 
propagates
         // to every request via `_attach_metadata` and the reattach stream.
         if let Some(tok) = builder.token() {
-            // Never send the bearer token in cleartext. The scheme above is 
chosen from
-            // `use_ssl()` alone, so without this guard 
`sc://host/;token=SECRET` (no
-            // `use_ssl`) would ship `Authorization: Bearer SECRET` over plain 
HTTP. The
-            // reference client cannot reach that state: `toChannel` sends a 
token only
-            // over `ssl_channel_credentials()` or, for `localhost`, the 
loopback
-            // `local_channel_credentials()`, and grpcio refuses call 
credentials on an
-            // insecure channel outright. Mirror that by requiring TLS for a 
token unless
-            // the endpoint is loopback; fail loudly rather than downgrade 
silently.
-            if !builder.use_ssl() && !builder.is_loopback() {
-                // A misconfigured connection string is a value error 
(surfaces as
-                // `PySparkValueError`), not a generic runtime failure.
-                return Err(SparkError::value_msg(format!(
-                    "Refusing to send the authentication token to '{}' over an 
insecure \
-                     connection. Set 'use_ssl=true' in the connection string 
to enable TLS \
-                     (a token is only allowed without TLS when connecting to 
localhost).",
-                    builder.host()
-                )));
-            }
+            // The token is sent over TLS, or over plaintext only on a 
loopback host: the
+            // scheme above is `builder.secure()`, which is TLS for a token on 
any other
+            // host, as the reference client does.
             metadata.push((AUTHORIZATION_HEADER.to_string(), format!("Bearer 
{}", tok)));
         }
 
@@ -355,7 +340,7 @@ impl SparkConnectClient {
             channel,
             stub,
             session_id,
-            user_id: builder.user_id().map(String::from),
+            user_id: 
builder.user_id().map(String::from).or_else(default_user_id),
             metadata,
             user_agent,
             user_context_extensions: Default::default(),
@@ -1552,6 +1537,21 @@ pub struct ReattachableResponseStream {
     released: bool,
 }
 
+/// The user ID of a connection string that does not set one: `SPARK_USER`, 
else `USER`, as
+/// `SparkConnectClient.__init__` in PySpark takes it.
+fn default_user_id() -> Option<String> {
+    user_id_from_env(std::env::var("SPARK_USER"), std::env::var("USER"))
+}
+
+/// The user ID from the environment, as `os.getenv("SPARK_USER", 
os.getenv("USER"))`: `USER`
+/// is only read when `SPARK_USER` is unset. An empty value is no user ID.
+fn user_id_from_env(
+    spark_user: std::result::Result<String, std::env::VarError>,
+    user: std::result::Result<String, std::env::VarError>,
+) -> Option<String> {
+    spark_user.or(user).ok().filter(|u| !u.is_empty())
+}
+
 impl SparkConnectClient {
     /// The gRPC metadata of every request about an execution.
     fn execution_metadata(&self) -> MetadataMap {
@@ -2072,20 +2072,98 @@ mod tests {
         );
     }
 
-    #[tokio::test]
-    async fn test_token_without_ssl_to_remote_host_is_rejected() {
-        // A token over a non-loopback host without `use_ssl=true` would ship 
the bearer
-        // credential in cleartext; the client must refuse rather than 
downgrade silently.
-        let builder = 
ChannelBuilder::parse("sc://example.com/;token=SECRET").unwrap();
-        match SparkConnectClient::connect(&builder).await {
-            Ok(_) => panic!("expected a token over cleartext to a remote host 
to be rejected"),
-            Err(e) => assert!(
-                e.to_string().contains("use_ssl=true"),
-                "expected the error to point at use_ssl=true, got: {e}"
-            ),
+    /// Connects a client on a runtime of its own, for the tests that set the 
environment.
+    /// The channel is lazy, so no server is dialed.
+    fn connect_blocking(url: &str) -> SparkConnectClient {
+        let builder = ChannelBuilder::parse(url).unwrap();
+        let runtime = tokio::runtime::Builder::new_current_thread()
+            .enable_all()
+            .build()
+            .unwrap();
+        runtime
+            .block_on(SparkConnectClient::connect(&builder))
+            .unwrap_or_else(|e| panic!("{url} should connect, got: {e}"))
+    }
+
+    fn has_bearer(client: &SparkConnectClient, token: &str) -> bool {
+        let bearer = format!("Bearer {token}");
+        client
+            .metadata
+            .iter()
+            .any(|(k, v)| k == "authorization" && *v == bearer)
+    }
+
+    #[test]
+    fn test_token_without_ssl_to_remote_host_is_sent_over_tls() {
+        // A token over a remote host is not refused: its channel is TLS, 
which the
+        // reference client also uses for a token on any host but localhost.
+        let _env = 
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+        for url in [
+            "sc://example.com/;token=SECRET",
+            "sc://10.0.0.1:443/;token=SECRET",
+        ] {
+            assert!(ChannelBuilder::parse(url).unwrap().secure(), "{url}");
+            let client = connect_blocking(url);
+            assert!(
+                has_bearer(&client, "SECRET"),
+                "{url}: {:?}",
+                client.metadata
+            );
         }
     }
 
+    #[test]
+    fn test_token_of_the_environment_to_remote_host_is_sent_over_tls() {
+        let env = 
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", Some("ENV"))]);
+        assert!(ChannelBuilder::parse("sc://example.com").unwrap().secure());
+        let client = connect_blocking("sc://example.com");
+        assert!(has_bearer(&client, "ENV"), "{:?}", client.metadata);
+        drop(env);
+        // Without a token, nothing is attached.
+        let _none = 
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+        let client = connect_blocking("sc://example.com");
+        assert!(!has_bearer(&client, "ENV"), "{:?}", client.metadata);
+    }
+
+    #[test]
+    fn test_user_id_from_the_environment_as_pyspark() {
+        let some = |s: &str| Ok::<String, std::env::VarError>(s.to_string());
+        let unset = || Err::<String, 
std::env::VarError>(std::env::VarError::NotPresent);
+        assert_eq!(
+            user_id_from_env(some("alice"), some("bob")),
+            Some("alice".into())
+        );
+        assert_eq!(user_id_from_env(unset(), some("bob")), Some("bob".into()));
+        // A `SPARK_USER` that is set to an empty string is the user ID, so 
`USER` is not read.
+        assert_eq!(user_id_from_env(some(""), some("bob")), None);
+        assert_eq!(user_id_from_env(unset(), some("")), None);
+        assert_eq!(user_id_from_env(unset(), unset()), None);
+    }
+
+    #[test]
+    fn test_user_id_of_the_connection_then_the_environment() {
+        // The connection string's user_id, else SPARK_USER, else USER, as 
PySpark does.
+        let user_id = |url: &str, spark_user: Option<&str>, user: 
Option<&str>| {
+            let _env = crate::channel::set_env(&[("SPARK_USER", spark_user), 
("USER", user)]);
+            let client = connect_blocking(url);
+            let id = client.user_id().map(String::from);
+            id
+        };
+        assert_eq!(
+            user_id("sc://localhost/;user_id=url", Some("alice"), 
Some("bob")).as_deref(),
+            Some("url")
+        );
+        assert_eq!(
+            user_id("sc://localhost", Some("alice"), Some("bob")).as_deref(),
+            Some("alice")
+        );
+        assert_eq!(
+            user_id("sc://localhost", None, Some("bob")).as_deref(),
+            Some("bob")
+        );
+        assert_eq!(user_id("sc://localhost", None, None), None);
+    }
+
     #[tokio::test]
     async fn test_token_without_ssl_to_localhost_is_allowed() {
         // Local development mirrors upstream's `local_channel_credentials()` 
exception:
diff --git a/crates/spark-connect-core/tests/user_context.rs 
b/crates/spark-connect-core/tests/user_context.rs
index 6dce9b7c..09c9831d 100644
--- a/crates/spark-connect-core/tests/user_context.rs
+++ b/crates/spark-connect-core/tests/user_context.rs
@@ -425,21 +425,22 @@ async fn 
typed_requests_take_a_missing_user_id_from_the_connection() {
 }
 
 #[tokio::test]
-async fn typed_requests_keep_their_user_context_without_a_configured_user_id() 
{
+async fn typed_requests_keep_their_user_context_with_an_empty_user_id() {
     let explicit = UserContext {
         user_id: "explicit".into(),
         user_name: "name".into(),
         extensions: vec![extension()],
     };
     for (given, expected) in [
-        // The message is still sent, as for every RPC that the client builds 
itself.
+        // The message is still sent, as for every RPC that the client builds 
itself. An
+        // empty `user_id` in the connection string is no user, whatever 
SPARK_USER or USER is.
         (None, UserContext::default()),
         (Some(UserContext::default()), UserContext::default()),
         (Some(explicit.clone()), explicit),
     ] {
         let fixture = Fixture::start([Execution::Complete, 
Execution::Complete]).await;
-        let client = fixture.client(None).await;
-        assert_eq!(client.user_id(), None);
+        let client = fixture.client(Some("")).await;
+        assert_eq!(client.user_id(), Some(""));
         for (method, context) in send_typed_requests(&fixture, &client, 
&given).await {
             assert_eq!(context.as_ref(), Some(&expected), "{method} for 
{given:?}");
         }
diff --git a/docs/configuration.md b/docs/configuration.md
index 58e25c32..486c8b1d 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -8,7 +8,7 @@ Connection strings follow the format 
`sc://host:port/;param=value`. Parameters a
 
 | Parameter | Purpose | Example |
 | --- | --- | --- |
-| `token` | Authentication token | `sc://host:15002/;token=abc123` |
+| `token` | Authentication token, sent over TLS to any host but localhost | 
`sc://host:15002/;token=abc123` |
 | `user_id` | User identity | `sc://host:15002/;user_id=alice` |
 | `session_id` | Session identifier | `sc://host:15002/;session_id=sess_xyz` |
 | `use_ssl` | Enable TLS | `sc://host:15002/;use_ssl=true` |
@@ -71,12 +71,12 @@ let spark = SparkSessionBuilder::default()
     .get_or_create()?;
 ```
 
-Replace `spark.example.com`, the port, and authentication parameters as 
needed. If TLS is required:
+Replace `spark.example.com`, the port, and authentication parameters as 
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token 
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same. 
Only to `localhost`, `127.0.0.1` and `[::1]` is a token sent without TLS, for 
local development. Without a token, `use_ssl=true` turns TLS on:
 
 ```rust
 use spark_connect::SparkSessionBuilder;
 
 let spark = SparkSessionBuilder::default()
-    .remote("sc://spark.example.com:15002/;use_ssl=true;token=YOUR_TOKEN")
+    .remote("sc://spark.example.com:15002/;use_ssl=true")
     .get_or_create()?;
 ```
diff --git a/python/tests/test_arrow_handoff.py 
b/python/tests/test_arrow_handoff.py
index 60563cd1..cfe35c1f 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -717,6 +717,8 @@ def 
test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch, u
 
     # Exercise ChannelBuilder's default prefix when no URL or environment 
value is set.
     monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+    # Without a user_id in the URL, the user ID is SPARK_USER, as PySpark's.
+    monkeypatch.setenv("SPARK_USER", "env-user")
     table = pa.table({"id": range(10_000)})
     schema = T.StructType([T.StructField("id", T.LongType())])
     with _arrow_input_service(protocol, user_id=user_id, 
user_agent=user_agent, **{
@@ -731,7 +733,7 @@ def 
test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch, u
         _assert_artifact_contexts(
             state["upload_contexts"] + state["status_contexts"],
             session_id=spark.session_id,
-            user_id=user_id or "",
+            user_id=user_id or "env-user",
             user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
         )
 
@@ -745,6 +747,8 @@ def test_file_upload_attribution_on_every_frame(
     protocol, monkeypatch, tmp_path, user_id, user_agent, upload
 ):
     monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+    # Without a user_id in the URL, the user ID is SPARK_USER, as PySpark's.
+    monkeypatch.setenv("SPARK_USER", "env-user")
     small, large = tmp_path / "small.txt", tmp_path / "large.txt"
     small.write_bytes(b"small artifact")
     chunk_size = 32 * 1024  # Mirrors CHUNK_SIZE in 
crates/spark-connect-core/src/artifact.rs.
@@ -776,7 +780,7 @@ def test_file_upload_attribution_on_every_frame(
         _assert_artifact_contexts(
             state["upload_contexts"],
             session_id=spark.session_id,
-            user_id=user_id or "",
+            user_id=user_id or "env-user",
             user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
         )
 
diff --git a/python/tests/test_dropin_offline.py 
b/python/tests/test_dropin_offline.py
index cdeca982..92b41e2c 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1131,6 +1131,29 @@ def 
test_repartition_by_range_without_a_count_sends_none_as_pyspark(mock_connect
     assert sent(df.repartitionByRange(2, "id")) == [2]
 
 
+def 
test_session_id_of_any_uuid_version_and_the_user_id_of_the_requests(monkeypatch,
 mock_connect):
+    # A session_id is any UUID, as uuid.UUID(session_id, version=4) takes it, 
and is sent as
+    # given. The user ID of each request is SPARK_USER, else USER, as PySpark 
sets it.
+    from pyspark.sql import SparkSession
+
+    _, service = mock_connect
+    session_id = "6ba7b810-9dad-11d1-80b4-00c04fd430c8"  # a version 1 UUID
+    for name, environment in (("SPARK_USER", "hook-user"), ("USER", 
"hook-os-user")):
+        monkeypatch.delenv("SPARK_USER", raising=False)
+        monkeypatch.delenv("USER", raising=False)
+        monkeypatch.setenv(name, environment)
+        other = 
SparkSession.builder.remote(f"{service.remote}/;session_id={session_id}").create()
+        try:
+            assert other.session_id == session_id
+            other.range(1).collect()
+            # ExecutePlanRequest.session_id=1, user_context=2; 
UserContext.user_id=1.
+            request = parse_message(service.execute_requests[-1])
+            assert request[1][0].decode() == session_id
+            assert parse_message(request[2][0])[1][0].decode() == environment
+        finally:
+            other.stop()
+
+
 def test_channel_builder_uses_endpoint():
     # channelBuilder reconstructs an sc:// URL from the builder's host/port (+ 
params)
     # for the native transport, returning a configured Builder (no longer 
raises).


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to