This is an automated email from the ASF dual-hosted git repository.

HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git


The following commit(s) were added to refs/heads/master by this push:
     new c329f4a0 [SPARK-61205][CONNECT] Use TLS for a token on any host other 
than localhost, also 127.0.0.1 and [::1], as PySpark does
c329f4a0 is described below

commit c329f4a037358e24446a4fd7099d1e523c29d802
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:04:47 2026 +0900

    [SPARK-61205][CONNECT] Use TLS for a token on any host other than 
localhost, also 127.0.0.1 and [::1], as PySpark does
    
    ### What changes were proposed in this pull request?
    
    [SPARK-61205](https://issues.apache.org/jira/browse/SPARK-61205): Use TLS 
for a token on any host other than `localhost`, also `127.0.0.1` and `[::1]`, 
as PySpark does.
    
    - `ChannelBuilder::secure()` is TLS when `use_ssl=true`, or when a token is 
present (in the connection string or in `SPARK_CONNECT_AUTHENTICATE_TOKEN`), 
unless the host is `localhost`. The loopback literals `127.0.0.1` and `[::1]` 
were exempt before, so a token to them was sent in cleartext over `http://`.
    - The exemption is now exactly the host `localhost`, compared 
case-insensitively, because `urllib` lowercases the host that PySpark compares 
(`core.py` `toChannel`: `not use_ssl and host == "localhost"`). `LOCALHOST` is 
therefore plaintext, as in PySpark, and `127.0.0.1` and `[::1]` are TLS.
    - The token guard that SPARK-59139 added uses the same decision, so a token 
is never sent in cleartext to a host other than `localhost`.
    - The TLS server name of an IPv6 host is its address without the brackets. 
A TLS channel to `[::1]` failed to configure before, because the bracketed name 
is not a valid server name.
    - `docs/configuration.md` says that a token is sent without TLS only to 
`localhost`.
    - The Rust `ChannelBuilder` gets `is_localhost()`, which `secure()` now 
uses. `is_loopback()` stays, with its meaning (localhost or a loopback 
literal), so the public API of `spark-connect-core` only grows.
    
    Official PySpark 4.3.0 against a plaintext server, for 
`sc://<host>:<port>/;token=T`: `localhost` and `LOCALHOST` connect, `127.0.0.1` 
and `[::1]` time out, because they use TLS. This PR gives the same results. The 
Python drop-in has no TLS decision of its own: it passes the URL to this code.
    
    ### Why are the changes needed?
    
    A connection string such as `sc://127.0.0.1:443/;token=T`, for example 
through an SSH tunnel to a TLS endpoint, made the client dial plaintext, and 
the token went in cleartext. PySpark uses TLS for every host except 
`localhost`, so the same URL works there and sends the token over TLS.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A token to `127.0.0.1` or `[::1]` without `use_ssl=true` is sent over 
TLS, as in PySpark. The connections to such a host without TLS now fail (a 
plaintext server times out). `localhost` is unchanged. The Rust API adds 
`ChannelBuilder::is_localhost()`.
    
    ### How was this patch tested?
    
    - Unit tests in `channel.rs`: 
`scheme_is_tls_for_a_token_on_any_host_but_localhost` checks the scheme for 
`localhost`, `LOCALHOST`, `127.0.0.1` and `[::1]` with and without a token, and 
with `use_ssl=true`. Without this change the `127.0.0.1`, `[::1]` and 
`LOCALHOST` rows fail.
    - `test_token_without_ssl_to_localhost_is_allowed` (client.rs) keeps 
`localhost` and `LOCALHOST` plaintext, with the bearer header attached. 
`test_token_to_loopback_ip_literals_is_sent_over_tls` connects to `127.0.0.1` 
and `[::1]` with a token, over TLS, and checks the bearer header. Without this 
change the scheme assertion fails; the IPv6 server name fix is what lets the 
TLS connect succeed.
    - The integration tests `deadlines.rs` and `reattach_cleanup.rs` connect to 
their in-process gRPC servers with a token. They now use 
`sc://localhost:<port>`, the plaintext case, because the server is plaintext.
    - Measured against a plaintext Spark Connect 4.3.0 server with the drop-in: 
`sc://localhost/;token=T` and `sc://LOCALHOST/;token=T` connect, 
`sc://127.0.0.1/;token=T` and `sc://[::1]/;token=T` time out (TLS), as official 
PySpark 4.3.0 does for the same URLs.
    - `cargo test -p apache-spark-connect-core --release` passes (all targets), 
and `cargo test -p apache-spark-connect --release --no-run` compiles all tests.
    - Against a local Spark 4.3.0 server, the live `test_dropin_e2e.py -k 
"remote or connect or channel or token or session"` tests (11 passed).
    - The lib tests of both crates (619 passed), the build of all tests, also 
with `wasm-udf`, and the offline Python suites (2539 passed, 8 skipped); 
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs 
the e2e and live suites.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    This pull request and its description were written by Isaac.
    
    Closes #391 from 
HyukjinKwon/SPARK-61205-r5-tls-for-a-token-except-on-localhost.
    
    Authored-by: Hyukjin Kwon <[email protected]>
    Signed-off-by: Hyukjin Kwon <[email protected]>
---
 crates/spark-connect-core/src/channel.rs           | 41 +++++++++++++---------
 crates/spark-connect-core/src/client.rs            | 32 +++++++++++++----
 crates/spark-connect-core/tests/deadlines.rs       |  3 +-
 .../spark-connect-core/tests/reattach_cleanup.rs   |  3 +-
 docs/configuration.md                              |  2 +-
 5 files changed, 54 insertions(+), 27 deletions(-)

diff --git a/crates/spark-connect-core/src/channel.rs 
b/crates/spark-connect-core/src/channel.rs
index 753fbba3..6f1045b2 100644
--- a/crates/spark-connect-core/src/channel.rs
+++ b/crates/spark-connect-core/src/channel.rs
@@ -157,23 +157,26 @@ impl ChannelBuilder {
     /// Whether the channel uses TLS, which is the scheme 
`SparkConnectClient::connect` dials.
     ///
     /// The reference client (`core.py` `ChannelBuilder.secure`) is TLS when 
`use_ssl` is set
-    /// or a token is present. A token on a loopback host stays plaintext 
here, as a
-    /// documented leniency for local development; the reference client uses 
local channel
-    /// credentials for `localhost` in that case. A token is therefore never 
sent in
-    /// cleartext to a remote host.
+    /// or a token is present, except for a token on `localhost`, which stays 
plaintext for
+    /// local development (`core.py` `toChannel` uses local channel 
credentials there). A
+    /// token is therefore never sent in cleartext to any other host, 
`127.0.0.1` and `[::1]`
+    /// included.
     pub fn secure(&self) -> bool {
-        self.use_ssl() || (self.token().is_some() && !self.is_loopback())
+        self.use_ssl() || (self.token().is_some() && !self.is_localhost())
     }
 
-    /// Whether the endpoint is a loopback address.
-    ///
-    /// The reference client (`core.py` `toChannel`) permits a token over an
-    /// unencrypted channel only for `host == "localhost"` (via
-    /// `grpc.local_channel_credentials()`); every other host is forced onto 
TLS.
-    /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost`, 
so
-    /// local development keeps working without `use_ssl=true`.
+    /// Whether the host is `localhost`. The reference client compares 
`localhost` after
+    /// `urllib` has lowercased the host (`core.py` `toChannel`), so the match 
is
+    /// case-insensitive. The IP loopback literals are not `localhost`, so 
they use TLS with
+    /// a token.
+    pub fn is_localhost(&self) -> bool {
+        self.host.eq_ignore_ascii_case("localhost")
+    }
+
+    /// Whether the endpoint is a loopback address: `localhost` or the 
IPv4/IPv6 loopback
+    /// literals. Only `localhost` keeps a token in plaintext; see 
[`Self::secure`].
     pub fn is_loopback(&self) -> bool {
-        matches!(self.host.as_str(), "localhost" | "127.0.0.1" | "[::1]")
+        self.is_localhost() || matches!(self.host.as_str(), "127.0.0.1" | 
"[::1]")
     }
 
     pub fn user_id(&self) -> Option<&str> {
@@ -438,9 +441,10 @@ mod tests {
     }
 
     #[test]
-    fn scheme_is_tls_for_a_token_on_a_remote_host() {
+    fn scheme_is_tls_for_a_token_on_any_host_but_localhost() {
         let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
-        // (url, TLS). A token goes over TLS to any host but a loopback one.
+        // (url, TLS). A token goes over TLS to any host but `localhost`, as 
PySpark does, so
+        // the loopback literals use TLS too. `use_ssl` is TLS for every host.
         for (url, tls) in [
             ("sc://example.com", false),
             ("sc://example.com/;use_ssl=true", true),
@@ -450,10 +454,13 @@ mod tests {
             ("sc://localhost/;token=T", false),
             ("sc://localhost/;use_ssl=true", true),
             ("sc://localhost/;use_ssl=true;token=T", true),
-            ("sc://127.0.0.1/;token=T", false),
+            ("sc://LOCALHOST/;token=T", false),
+            ("sc://127.0.0.1/;token=T", true),
             ("sc://127.0.0.1/;use_ssl=true", true),
-            ("sc://[::1]:15002/;token=T", false),
+            ("sc://127.0.0.1/", false),
+            ("sc://[::1]:15002/;token=T", true),
             ("sc://[::1]:15002/;use_ssl=true", true),
+            ("sc://[::1]:15002/", false),
         ] {
             let c = ChannelBuilder::parse(url).unwrap();
             assert_eq!(c.secure(), tls, "{url}");
diff --git a/crates/spark-connect-core/src/client.rs 
b/crates/spark-connect-core/src/client.rs
index 9cb0ba5c..d8bdcba0 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -320,8 +320,10 @@ impl SparkConnectClient {
             // `with_native_roots()` is required: tonic >=0.11 does NOT load a 
trust store
             // from the `tls-native-roots` feature alone, so without this the 
TLS handshake
             // has no CA anchors, fails, and the retry policy masks it as a 
hang.
+            // The server name of an IPv6 host is its address without the 
brackets of the URL.
+            let domain = 
builder.host().trim_start_matches('[').trim_end_matches(']');
             let tls_config = ClientTlsConfig::new()
-                .domain_name(builder.host())
+                .domain_name(domain)
                 .with_native_roots();
 
             
tonic::transport::Channel::from_shared(format!("https://{endpoint}";))
@@ -352,9 +354,9 @@ impl SparkConnectClient {
         // excluded from `metadata()`, so attach it explicitly here; this then 
propagates
         // to every request via `_attach_metadata` and the reattach stream.
         if let Some(tok) = builder.token() {
-            // The token is sent over TLS, or over plaintext only on a 
loopback host: the
-            // scheme above is `builder.secure()`, which is TLS for a token on 
any other
-            // host, as the reference client does.
+            // The token is sent over TLS, or over plaintext only on 
`localhost`: the scheme
+            // above is `builder.secure()`, which is TLS for a token on any 
other host, as
+            // the reference client does.
             metadata.push((AUTHORIZATION_HEADER.to_string(), format!("Bearer 
{}", tok)));
         }
 
@@ -2350,13 +2352,13 @@ mod tests {
     #[tokio::test]
     async fn test_token_without_ssl_to_localhost_is_allowed() {
         // Local development mirrors upstream's `local_channel_credentials()` 
exception:
-        // a token over plaintext loopback is permitted and the header is 
attached.
+        // a token over plaintext `localhost` is permitted and the header is 
attached.
         for url in [
             "sc://localhost/;token=SECRET",
-            "sc://127.0.0.1/;token=SECRET",
-            "sc://[::1]/;token=SECRET",
+            "sc://LOCALHOST/;token=SECRET",
         ] {
             let builder = ChannelBuilder::parse(url).unwrap();
+            assert!(!builder.secure(), "{url}");
             let client = SparkConnectClient::connect(&builder)
                 .await
                 .unwrap_or_else(|e| panic!("{url} should be allowed, got: 
{e}"));
@@ -2371,6 +2373,22 @@ mod tests {
         }
     }
 
+    #[tokio::test]
+    async fn test_token_to_loopback_ip_literals_is_sent_over_tls() {
+        // Only `localhost` is exempt: the IPv4 and IPv6 loopback literals are 
TLS with a
+        // token, as in PySpark, so the channel is https and the header is 
still attached.
+        for url in ["sc://127.0.0.1/;token=SECRET", 
"sc://[::1]/;token=SECRET"] {
+            let builder = ChannelBuilder::parse(url).unwrap();
+            assert!(builder.secure(), "{url}");
+            let client = SparkConnectClient::connect(&builder).await.unwrap();
+            assert!(
+                has_bearer(&client, "SECRET"),
+                "{url}: {:?}",
+                client.metadata
+            );
+        }
+    }
+
     #[tokio::test]
     async fn test_get_configs_request_structure() {
         let builder = ChannelBuilder::parse("sc://localhost").unwrap();
diff --git a/crates/spark-connect-core/tests/deadlines.rs 
b/crates/spark-connect-core/tests/deadlines.rs
index 878cc4e4..6fbedc2d 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -155,7 +155,8 @@ impl Fixture {
                 .unwrap();
         });
         let builder = ChannelBuilder::parse(&format!(
-            
"sc://{address}/;user_id=transport-user;token=fixture-token;session_id=00000000-0000-4000-8000-000000000001"
+            
"sc://localhost:{}/;user_id=transport-user;token=fixture-token;session_id=00000000-0000-4000-8000-000000000001",
+            address.port()
         ))
         .unwrap();
         let mut client = bounded(SparkConnectClient::connect(&builder))
diff --git a/crates/spark-connect-core/tests/reattach_cleanup.rs 
b/crates/spark-connect-core/tests/reattach_cleanup.rs
index 3fa59fdb..cdc9a9e8 100644
--- a/crates/spark-connect-core/tests/reattach_cleanup.rs
+++ b/crates/spark-connect-core/tests/reattach_cleanup.rs
@@ -159,7 +159,8 @@ impl Fixture {
                 .unwrap();
         });
         let builder = ChannelBuilder::parse(&format!(
-            
"sc://{address}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value;session_id=00000000-0000-4000-8000-000000000001"
+            
"sc://localhost:{}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value;session_id=00000000-0000-4000-8000-000000000001",
+            address.port()
         ))
         .unwrap();
         let mut client = bounded(SparkConnectClient::connect(&builder))
diff --git a/docs/configuration.md b/docs/configuration.md
index 486c8b1d..37dcd77e 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -71,7 +71,7 @@ let spark = SparkSessionBuilder::default()
     .get_or_create()?;
 ```
 
-Replace `spark.example.com`, the port, and authentication parameters as 
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token 
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same. 
Only to `localhost`, `127.0.0.1` and `[::1]` is a token sent without TLS, for 
local development. Without a token, `use_ssl=true` turns TLS on:
+Replace `spark.example.com`, the port, and authentication parameters as 
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token 
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same. 
Only to `localhost` is a token sent without TLS, for local development; to 
`127.0.0.1` and `[::1]` it is sent over TLS, as in PySpark. Without a token, 
`use_ssl=true` turns TLS on:
 
 ```rust
 use spark_connect::SparkSessionBuilder;


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to