This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new c329f4a0 [SPARK-61205][CONNECT] Use TLS for a token on any host other
than localhost, also 127.0.0.1 and [::1], as PySpark does
c329f4a0 is described below
commit c329f4a037358e24446a4fd7099d1e523c29d802
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 10:04:47 2026 +0900
[SPARK-61205][CONNECT] Use TLS for a token on any host other than
localhost, also 127.0.0.1 and [::1], as PySpark does
### What changes were proposed in this pull request?
[SPARK-61205](https://issues.apache.org/jira/browse/SPARK-61205): Use TLS
for a token on any host other than `localhost`, also `127.0.0.1` and `[::1]`,
as PySpark does.
- `ChannelBuilder::secure()` is TLS when `use_ssl=true`, or when a token is
present (in the connection string or in `SPARK_CONNECT_AUTHENTICATE_TOKEN`),
unless the host is `localhost`. The loopback literals `127.0.0.1` and `[::1]`
were exempt before, so a token to them was sent in cleartext over `http://`.
- The exemption is now exactly the host `localhost`, compared
case-insensitively, because `urllib` lowercases the host that PySpark compares
(`core.py` `toChannel`: `not use_ssl and host == "localhost"`). `LOCALHOST` is
therefore plaintext, as in PySpark, and `127.0.0.1` and `[::1]` are TLS.
- The token guard that SPARK-59139 added uses the same decision, so a token
is never sent in cleartext to a host other than `localhost`.
- The TLS server name of an IPv6 host is its address without the brackets.
A TLS channel to `[::1]` failed to configure before, because the bracketed name
is not a valid server name.
- `docs/configuration.md` says that a token is sent without TLS only to
`localhost`.
- The Rust `ChannelBuilder` gets `is_localhost()`, which `secure()` now
uses. `is_loopback()` stays, with its meaning (localhost or a loopback
literal), so the public API of `spark-connect-core` only grows.
Official PySpark 4.3.0 against a plaintext server, for
`sc://<host>:<port>/;token=T`: `localhost` and `LOCALHOST` connect, `127.0.0.1`
and `[::1]` time out, because they use TLS. This PR gives the same results. The
Python drop-in has no TLS decision of its own: it passes the URL to this code.
### Why are the changes needed?
A connection string such as `sc://127.0.0.1:443/;token=T`, for example
through an SSH tunnel to a TLS endpoint, made the client dial plaintext, and
the token went in cleartext. PySpark uses TLS for every host except
`localhost`, so the same URL works there and sends the token over TLS.
### Does this PR introduce _any_ user-facing change?
Yes. A token to `127.0.0.1` or `[::1]` without `use_ssl=true` is sent over
TLS, as in PySpark. The connections to such a host without TLS now fail (a
plaintext server times out). `localhost` is unchanged. The Rust API adds
`ChannelBuilder::is_localhost()`.
### How was this patch tested?
- Unit tests in `channel.rs`:
`scheme_is_tls_for_a_token_on_any_host_but_localhost` checks the scheme for
`localhost`, `LOCALHOST`, `127.0.0.1` and `[::1]` with and without a token, and
with `use_ssl=true`. Without this change the `127.0.0.1`, `[::1]` and
`LOCALHOST` rows fail.
- `test_token_without_ssl_to_localhost_is_allowed` (client.rs) keeps
`localhost` and `LOCALHOST` plaintext, with the bearer header attached.
`test_token_to_loopback_ip_literals_is_sent_over_tls` connects to `127.0.0.1`
and `[::1]` with a token, over TLS, and checks the bearer header. Without this
change the scheme assertion fails; the IPv6 server name fix is what lets the
TLS connect succeed.
- The integration tests `deadlines.rs` and `reattach_cleanup.rs` connect to
their in-process gRPC servers with a token. They now use
`sc://localhost:<port>`, the plaintext case, because the server is plaintext.
- Measured against a plaintext Spark Connect 4.3.0 server with the drop-in:
`sc://localhost/;token=T` and `sc://LOCALHOST/;token=T` connect,
`sc://127.0.0.1/;token=T` and `sc://[::1]/;token=T` time out (TLS), as official
PySpark 4.3.0 does for the same URLs.
- `cargo test -p apache-spark-connect-core --release` passes (all targets),
and `cargo test -p apache-spark-connect --release --no-run` compiles all tests.
- Against a local Spark 4.3.0 server, the live `test_dropin_e2e.py -k
"remote or connect or channel or token or session"` tests (11 passed).
- The lib tests of both crates (619 passed), the build of all tests, also
with `wasm-udf`, and the offline Python suites (2539 passed, 8 skipped);
rustfmt passes, and clippy reports no diagnostics in the changed lines. CI runs
the e2e and live suites.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #391 from
HyukjinKwon/SPARK-61205-r5-tls-for-a-token-except-on-localhost.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/channel.rs | 41 +++++++++++++---------
crates/spark-connect-core/src/client.rs | 32 +++++++++++++----
crates/spark-connect-core/tests/deadlines.rs | 3 +-
.../spark-connect-core/tests/reattach_cleanup.rs | 3 +-
docs/configuration.md | 2 +-
5 files changed, 54 insertions(+), 27 deletions(-)
diff --git a/crates/spark-connect-core/src/channel.rs
b/crates/spark-connect-core/src/channel.rs
index 753fbba3..6f1045b2 100644
--- a/crates/spark-connect-core/src/channel.rs
+++ b/crates/spark-connect-core/src/channel.rs
@@ -157,23 +157,26 @@ impl ChannelBuilder {
/// Whether the channel uses TLS, which is the scheme
`SparkConnectClient::connect` dials.
///
/// The reference client (`core.py` `ChannelBuilder.secure`) is TLS when
`use_ssl` is set
- /// or a token is present. A token on a loopback host stays plaintext
here, as a
- /// documented leniency for local development; the reference client uses
local channel
- /// credentials for `localhost` in that case. A token is therefore never
sent in
- /// cleartext to a remote host.
+ /// or a token is present, except for a token on `localhost`, which stays
plaintext for
+ /// local development (`core.py` `toChannel` uses local channel
credentials there). A
+ /// token is therefore never sent in cleartext to any other host,
`127.0.0.1` and `[::1]`
+ /// included.
pub fn secure(&self) -> bool {
- self.use_ssl() || (self.token().is_some() && !self.is_loopback())
+ self.use_ssl() || (self.token().is_some() && !self.is_localhost())
}
- /// Whether the endpoint is a loopback address.
- ///
- /// The reference client (`core.py` `toChannel`) permits a token over an
- /// unencrypted channel only for `host == "localhost"` (via
- /// `grpc.local_channel_credentials()`); every other host is forced onto
TLS.
- /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost`,
so
- /// local development keeps working without `use_ssl=true`.
+ /// Whether the host is `localhost`. The reference client compares
`localhost` after
+ /// `urllib` has lowercased the host (`core.py` `toChannel`), so the match
is
+ /// case-insensitive. The IP loopback literals are not `localhost`, so
they use TLS with
+ /// a token.
+ pub fn is_localhost(&self) -> bool {
+ self.host.eq_ignore_ascii_case("localhost")
+ }
+
+ /// Whether the endpoint is a loopback address: `localhost` or the
IPv4/IPv6 loopback
+ /// literals. Only `localhost` keeps a token in plaintext; see
[`Self::secure`].
pub fn is_loopback(&self) -> bool {
- matches!(self.host.as_str(), "localhost" | "127.0.0.1" | "[::1]")
+ self.is_localhost() || matches!(self.host.as_str(), "127.0.0.1" |
"[::1]")
}
pub fn user_id(&self) -> Option<&str> {
@@ -438,9 +441,10 @@ mod tests {
}
#[test]
- fn scheme_is_tls_for_a_token_on_a_remote_host() {
+ fn scheme_is_tls_for_a_token_on_any_host_but_localhost() {
let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
- // (url, TLS). A token goes over TLS to any host but a loopback one.
+ // (url, TLS). A token goes over TLS to any host but `localhost`, as
PySpark does, so
+ // the loopback literals use TLS too. `use_ssl` is TLS for every host.
for (url, tls) in [
("sc://example.com", false),
("sc://example.com/;use_ssl=true", true),
@@ -450,10 +454,13 @@ mod tests {
("sc://localhost/;token=T", false),
("sc://localhost/;use_ssl=true", true),
("sc://localhost/;use_ssl=true;token=T", true),
- ("sc://127.0.0.1/;token=T", false),
+ ("sc://LOCALHOST/;token=T", false),
+ ("sc://127.0.0.1/;token=T", true),
("sc://127.0.0.1/;use_ssl=true", true),
- ("sc://[::1]:15002/;token=T", false),
+ ("sc://127.0.0.1/", false),
+ ("sc://[::1]:15002/;token=T", true),
("sc://[::1]:15002/;use_ssl=true", true),
+ ("sc://[::1]:15002/", false),
] {
let c = ChannelBuilder::parse(url).unwrap();
assert_eq!(c.secure(), tls, "{url}");
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 9cb0ba5c..d8bdcba0 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -320,8 +320,10 @@ impl SparkConnectClient {
// `with_native_roots()` is required: tonic >=0.11 does NOT load a
trust store
// from the `tls-native-roots` feature alone, so without this the
TLS handshake
// has no CA anchors, fails, and the retry policy masks it as a
hang.
+ // The server name of an IPv6 host is its address without the
brackets of the URL.
+ let domain =
builder.host().trim_start_matches('[').trim_end_matches(']');
let tls_config = ClientTlsConfig::new()
- .domain_name(builder.host())
+ .domain_name(domain)
.with_native_roots();
tonic::transport::Channel::from_shared(format!("https://{endpoint}"))
@@ -352,9 +354,9 @@ impl SparkConnectClient {
// excluded from `metadata()`, so attach it explicitly here; this then
propagates
// to every request via `_attach_metadata` and the reattach stream.
if let Some(tok) = builder.token() {
- // The token is sent over TLS, or over plaintext only on a
loopback host: the
- // scheme above is `builder.secure()`, which is TLS for a token on
any other
- // host, as the reference client does.
+ // The token is sent over TLS, or over plaintext only on
`localhost`: the scheme
+ // above is `builder.secure()`, which is TLS for a token on any
other host, as
+ // the reference client does.
metadata.push((AUTHORIZATION_HEADER.to_string(), format!("Bearer
{}", tok)));
}
@@ -2350,13 +2352,13 @@ mod tests {
#[tokio::test]
async fn test_token_without_ssl_to_localhost_is_allowed() {
// Local development mirrors upstream's `local_channel_credentials()`
exception:
- // a token over plaintext loopback is permitted and the header is
attached.
+ // a token over plaintext `localhost` is permitted and the header is
attached.
for url in [
"sc://localhost/;token=SECRET",
- "sc://127.0.0.1/;token=SECRET",
- "sc://[::1]/;token=SECRET",
+ "sc://LOCALHOST/;token=SECRET",
] {
let builder = ChannelBuilder::parse(url).unwrap();
+ assert!(!builder.secure(), "{url}");
let client = SparkConnectClient::connect(&builder)
.await
.unwrap_or_else(|e| panic!("{url} should be allowed, got:
{e}"));
@@ -2371,6 +2373,22 @@ mod tests {
}
}
+ #[tokio::test]
+ async fn test_token_to_loopback_ip_literals_is_sent_over_tls() {
+ // Only `localhost` is exempt: the IPv4 and IPv6 loopback literals are
TLS with a
+ // token, as in PySpark, so the channel is https and the header is
still attached.
+ for url in ["sc://127.0.0.1/;token=SECRET",
"sc://[::1]/;token=SECRET"] {
+ let builder = ChannelBuilder::parse(url).unwrap();
+ assert!(builder.secure(), "{url}");
+ let client = SparkConnectClient::connect(&builder).await.unwrap();
+ assert!(
+ has_bearer(&client, "SECRET"),
+ "{url}: {:?}",
+ client.metadata
+ );
+ }
+ }
+
#[tokio::test]
async fn test_get_configs_request_structure() {
let builder = ChannelBuilder::parse("sc://localhost").unwrap();
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
index 878cc4e4..6fbedc2d 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -155,7 +155,8 @@ impl Fixture {
.unwrap();
});
let builder = ChannelBuilder::parse(&format!(
-
"sc://{address}/;user_id=transport-user;token=fixture-token;session_id=00000000-0000-4000-8000-000000000001"
+
"sc://localhost:{}/;user_id=transport-user;token=fixture-token;session_id=00000000-0000-4000-8000-000000000001",
+ address.port()
))
.unwrap();
let mut client = bounded(SparkConnectClient::connect(&builder))
diff --git a/crates/spark-connect-core/tests/reattach_cleanup.rs
b/crates/spark-connect-core/tests/reattach_cleanup.rs
index 3fa59fdb..cdc9a9e8 100644
--- a/crates/spark-connect-core/tests/reattach_cleanup.rs
+++ b/crates/spark-connect-core/tests/reattach_cleanup.rs
@@ -159,7 +159,8 @@ impl Fixture {
.unwrap();
});
let builder = ChannelBuilder::parse(&format!(
-
"sc://{address}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value;session_id=00000000-0000-4000-8000-000000000001"
+
"sc://localhost:{}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value;session_id=00000000-0000-4000-8000-000000000001",
+ address.port()
))
.unwrap();
let mut client = bounded(SparkConnectClient::connect(&builder))
diff --git a/docs/configuration.md b/docs/configuration.md
index 486c8b1d..37dcd77e 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -71,7 +71,7 @@ let spark = SparkSessionBuilder::default()
.get_or_create()?;
```
-Replace `spark.example.com`, the port, and authentication parameters as
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same.
Only to `localhost`, `127.0.0.1` and `[::1]` is a token sent without TLS, for
local development. Without a token, `use_ssl=true` turns TLS on:
+Replace `spark.example.com`, the port, and authentication parameters as
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same.
Only to `localhost` is a token sent without TLS, for local development; to
`127.0.0.1` and `[::1]` it is sent over TLS, as in PySpark. Without a token,
`use_ssl=true` turns TLS on:
```rust
use spark_connect::SparkSessionBuilder;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]