This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new cf0d2daf [SPARK-61182][CONNECT] Connect over TLS when the connection
string has a token for a host other than localhost, take any UUID as the
session ID, and default the user ID to SPARK_USER or USER, as PySpark does
cf0d2daf is described below
commit cf0d2dafcf94e008e823f8a082ca1c6e17ca4f2e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Sun Nov 8 20:29:44 2026 +0900
[SPARK-61182][CONNECT] Connect over TLS when the connection string has a
token for a host other than localhost, take any UUID as the session ID, and
default the user ID to SPARK_USER or USER, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61182](https://issues.apache.org/jira/browse/SPARK-61182): PySpark
connects over TLS when `use_ssl` is set or a token is present, a token on
`localhost` stays on local channel credentials, and a session ID is any UUID.
The user ID is the connection string's `user_id`, else `SPARK_USER`, else
`USER`.
- `ChannelBuilder::secure()` is the scheme predicate, and
`SparkConnectClient::connect` dials TLS when it holds. It is true for
`use_ssl=true`, and for a token on a host that is not loopback. A token on
`localhost`, `127.0.0.1` or `[::1]` stays plaintext, as before. The refusal
"Refusing to send the authentication token ... over an insecure connection" is
removed, since the token now always goes over TLS unless the host is loopback.
- `ChannelBuilder::session_id()` accepts any UUID that parses, and returns
the string as given. Malformed strings still raise `INVALID_SESSION_UUID_ID`.
- `SparkConnectClient::connect` takes the user ID from `SPARK_USER`, else
`USER`, when the connection string sets no `user_id`. An empty value is no user
ID. The user ID is then set in the user context of each request, as before for
an explicit `user_id`.
- `docs/configuration.md` says that a token implies TLS, and that a token
is sent without TLS only to `localhost`, `127.0.0.1` and `[::1]`.
### Why are the changes needed?
- A token for a host other than localhost, without `use_ssl=true`, failed
to connect. A call such as `sc://example.com:443/;token=T`, or a token from
`SPARK_CONNECT_AUTHENTICATE_TOKEN`, returned the refusal. PySpark connects over
TLS for it. `DefaultChannelBuilder(...).secure` in PySpark 4.3.0 returns `True`
for `sc://example.com/;token=T`, and for `sc://localhost/;token=T`.
- A UUID of version 1, such as
`sc://localhost/;session_id=6ba7b810-9dad-11d1-80b4-00c04fd430c8`, raised
`INVALID_SESSION_UUID_ID`, since the check required version 4. PySpark accepts
it: `DefaultChannelBuilder(...).session_id` returns the string.
- Without `user_id` in the connection string, the user context of every
request had an empty `user_id`. PySpark sends the OS user. With
`SPARK_USER=hook-user`, `SparkConnectClient(...)._user_id` is `'hook-user'` in
PySpark 4.3.0, and with only `USER=hook-os-user` it is `'hook-os-user'`.
### Does this PR introduce _any_ user-facing change?
Yes.
- A token for a host other than localhost is sent over TLS, and no longer
refused without `use_ssl=true`.
- A session ID of any UUID version is accepted, and sent as given.
- Requests carry the user ID from `SPARK_USER`, else `USER`, when the
connection string has no `user_id`. The same user ID is also sent as the
`user_id` gRPC header, which the client already sent for an explicit `user_id`.
- Known difference from PySpark: a token on `127.0.0.1` or `[::1]` stays
plaintext, where PySpark uses TLS for those hosts. This is the existing
loopback leniency, unchanged.
### How was this patch tested?
- Rust unit tests in `crates/spark-connect-core/src/channel.rs`:
`scheme_is_tls_for_a_token_on_a_remote_host`,
`scheme_uses_the_token_of_the_environment`,
`session_id_accepts_a_uuid_of_any_version_as_given`,
`session_id_rejects_a_malformed_uuid`.
- Rust unit tests in `crates/spark-connect-core/src/client.rs`:
`test_token_without_ssl_to_remote_host_is_sent_over_tls` (replaces the refusal
test), `test_token_of_the_environment_to_remote_host_is_sent_over_tls`,
`test_user_id_from_the_environment_as_pyspark`,
`test_user_id_of_the_connection_then_the_environment`.
- `crates/spark-connect-core/tests/user_context.rs`:
`typed_requests_keep_their_user_context_with_an_empty_user_id` now uses
`user_id=` in the connection string, since the default would otherwise read the
`USER` of the machine.
- `test_cached_arrow_upload_attribution_on_every_frame` and
`test_file_upload_attribution_on_every_frame` in
`python/tests/test_arrow_handoff.py` set `SPARK_USER` for a connection string
without `user_id`, and expect it as the user ID of every frame, where they
expected an empty one.
- Environment-variable tests take a shared lock (`channel::set_env`), so
they do not race other tests.
- Python offline test
`test_session_id_of_any_uuid_version_and_the_user_id_of_the_requests` in
`python/tests/test_dropin_offline.py`: a version 1 session ID is kept as given,
and the `ExecutePlanRequest` seen by the mock server has that session ID and
the user ID from `SPARK_USER`, then from `USER`.
- The same cases checked against official PySpark 4.3.0 with
`DefaultChannelBuilder` and `SparkConnectClient`: the session ID is accepted,
`secure` is as above, and the user ID is `SPARK_USER`, else `USER`.
- The Python test fails without this change, with `INVALID_SESSION_UUID_ID`.
- The lib tests of both crates (604 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2228 passed, 8 skipped), and the
live drop-in and Arrow suites (209 passed) and live cases (10 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #373 from HyukjinKwon/SPARK-61182-channel-builder-parity.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/channel.rs | 132 ++++++++++++++++++----
crates/spark-connect-core/src/client.rs | 140 ++++++++++++++++++------
crates/spark-connect-core/tests/user_context.rs | 9 +-
docs/configuration.md | 6 +-
python/tests/test_arrow_handoff.py | 8 +-
python/tests/test_dropin_offline.py | 23 ++++
6 files changed, 255 insertions(+), 63 deletions(-)
diff --git a/crates/spark-connect-core/src/channel.rs
b/crates/spark-connect-core/src/channel.rs
index 43a9e28b..753fbba3 100644
--- a/crates/spark-connect-core/src/channel.rs
+++ b/crates/spark-connect-core/src/channel.rs
@@ -154,14 +154,15 @@ impl ChannelBuilder {
.or_else(|| std::env::var("SPARK_CONNECT_AUTHENTICATE_TOKEN").ok())
}
- /// Mirrors the reference client's `secure` property (`use_ssl` OR a token
is
- /// present). NOTE: this does NOT gate the transport. It reads like a
security
- /// check but grants none: the token half was the source of the
cleartext-token
- /// bug, since a token alone reports "secure" while the scheme stays
plaintext.
- /// The real gate on sending the token is `use_ssl() && !is_loopback()` in
- /// `SparkConnectClient::connect`; this method is kept only for
parity/inspection.
+ /// Whether the channel uses TLS, which is the scheme
`SparkConnectClient::connect` dials.
+ ///
+ /// The reference client (`core.py` `ChannelBuilder.secure`) is TLS when
`use_ssl` is set
+ /// or a token is present. A token on a loopback host stays plaintext
here, as a
+ /// documented leniency for local development; the reference client uses
local channel
+ /// credentials for `localhost` in that case. A token is therefore never
sent in
+ /// cleartext to a remote host.
pub fn secure(&self) -> bool {
- self.use_ssl() || self.token().is_some()
+ self.use_ssl() || (self.token().is_some() && !self.is_loopback())
}
/// Whether the endpoint is a loopback address.
@@ -169,7 +170,7 @@ impl ChannelBuilder {
/// The reference client (`core.py` `toChannel`) permits a token over an
/// unencrypted channel only for `host == "localhost"` (via
/// `grpc.local_channel_credentials()`); every other host is forced onto
TLS.
- /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost`
so
+ /// We treat the IPv4/IPv6 loopback literals as equivalent to `localhost`,
so
/// local development keeps working without `use_ssl=true`.
pub fn is_loopback(&self) -> bool {
matches!(self.host.as_str(), "localhost" | "127.0.0.1" | "[::1]")
@@ -179,13 +180,14 @@ impl ChannelBuilder {
self.get(PARAM_USER_ID)
}
- /// Validates the `session_id` param is a v4 UUID (mirrors the Python
check).
+ /// Validates the `session_id` param is a UUID of any version, and returns
it as given.
+ /// As `uuid.UUID(session_id, version=4)` in PySpark, which does not check
the version.
pub fn session_id(&self) -> Result<Option<String>> {
match self.params.get(PARAM_SESSION_ID) {
None => Ok(None),
Some(s) => match uuid::Uuid::parse_str(s) {
- Ok(u) if u.get_version_num() == 4 => Ok(Some(s.clone())),
- Ok(_) | Err(_) => Err(SparkError::value(
+ Ok(_) => Ok(Some(s.clone())),
+ Err(_) => Err(SparkError::value(
"INVALID_SESSION_UUID_ID",
&[("arg_name", "session_id"), ("origin", "invalid UUID")],
)),
@@ -360,12 +362,52 @@ fn quote_len(s: &str) -> usize {
s.bytes().map(|b| if unreserved(b) { 1 } else { 3 }).sum()
}
+/// Serializes the tests that set or read the environment variables that the
channel reads,
+/// since the test threads share one environment.
+#[cfg(test)]
+pub(crate) static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(());
+
+/// Sets environment variables for one test, under `ENV_LOCK`, and restores
them on drop.
+/// A `None` value removes the variable.
+#[cfg(test)]
+pub(crate) struct EnvGuard {
+ _lock: std::sync::MutexGuard<'static, ()>,
+ saved: Vec<(&'static str, Option<std::ffi::OsString>)>,
+}
+
+#[cfg(test)]
+pub(crate) fn set_env(vars: &[(&'static str, Option<&str>)]) -> EnvGuard {
+ let lock = ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner());
+ let mut saved = Vec::new();
+ for (key, value) in vars {
+ saved.push((*key, std::env::var_os(key)));
+ match value {
+ Some(v) => std::env::set_var(key, v),
+ None => std::env::remove_var(key),
+ }
+ }
+ EnvGuard { _lock: lock, saved }
+}
+
+#[cfg(test)]
+impl Drop for EnvGuard {
+ fn drop(&mut self) {
+ for (key, value) in self.saved.iter().rev() {
+ match value {
+ Some(v) => std::env::set_var(key, v),
+ None => std::env::remove_var(key),
+ }
+ }
+ }
+}
+
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn basic_host_default_port() {
+ let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
let c = ChannelBuilder::parse("sc://localhost").unwrap();
assert_eq!(c.endpoint(), "localhost:15002");
assert!(!c.secure());
@@ -396,10 +438,40 @@ mod tests {
}
#[test]
- fn token_implies_secure() {
- let c = ChannelBuilder::parse("sc://localhost/;token=xyz").unwrap();
- assert!(c.secure());
- assert!(!c.use_ssl());
+ fn scheme_is_tls_for_a_token_on_a_remote_host() {
+ let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+ // (url, TLS). A token goes over TLS to any host but a loopback one.
+ for (url, tls) in [
+ ("sc://example.com", false),
+ ("sc://example.com/;use_ssl=true", true),
+ ("sc://example.com/;token=T", true),
+ ("sc://example.com/;use_ssl=false;token=T", true),
+ ("sc://10.0.0.1:443/;token=T", true),
+ ("sc://localhost/;token=T", false),
+ ("sc://localhost/;use_ssl=true", true),
+ ("sc://localhost/;use_ssl=true;token=T", true),
+ ("sc://127.0.0.1/;token=T", false),
+ ("sc://127.0.0.1/;use_ssl=true", true),
+ ("sc://[::1]:15002/;token=T", false),
+ ("sc://[::1]:15002/;use_ssl=true", true),
+ ] {
+ let c = ChannelBuilder::parse(url).unwrap();
+ assert_eq!(c.secure(), tls, "{url}");
+ }
+ }
+
+ #[test]
+ fn scheme_uses_the_token_of_the_environment() {
+ let _env = set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN",
Some("ENV"))]);
+ let remote = ChannelBuilder::parse("sc://example.com").unwrap();
+ assert_eq!(remote.token().as_deref(), Some("ENV"));
+ assert!(remote.secure());
+ let local = ChannelBuilder::parse("sc://localhost").unwrap();
+ assert_eq!(local.token().as_deref(), Some("ENV"));
+ assert!(!local.secure());
+ // The token of the connection string wins over the environment.
+ let explicit =
ChannelBuilder::parse("sc://example.com/;token=T").unwrap();
+ assert_eq!(explicit.token().as_deref(), Some("T"));
}
#[test]
@@ -447,13 +519,27 @@ mod tests {
}
#[test]
- fn session_id_must_be_uuid4() {
- let c = ChannelBuilder::parse(
- "sc://localhost/;session_id=550e8400-e29b-41d4-a716-446655440000",
- )
- .unwrap();
- assert!(c.session_id().is_ok());
- let bad =
ChannelBuilder::parse("sc://localhost/;session_id=not-a-uuid").unwrap();
- assert!(bad.session_id().is_err());
+ fn session_id_accepts_a_uuid_of_any_version_as_given() {
+ // Versions 1, 4, 7 and the nil UUID, as `uuid.UUID(session_id,
version=4)` accepts.
+ for id in [
+ "6ba7b810-9dad-11d1-80b4-00c04fd430c8",
+ "550e8400-e29b-41d4-a716-446655440000",
+ "018f0b8e-0000-7000-8000-000000000000",
+ "00000000-0000-0000-0000-000000000000",
+ ] {
+ let c =
ChannelBuilder::parse(&format!("sc://localhost/;session_id={id}")).unwrap();
+ assert_eq!(c.session_id().unwrap().as_deref(), Some(id));
+ }
+ let none = ChannelBuilder::parse("sc://localhost").unwrap();
+ assert_eq!(none.session_id().unwrap(), None);
+ }
+
+ #[test]
+ fn session_id_rejects_a_malformed_uuid() {
+ for id in ["not-a-uuid", "550e8400-e29b-41d4-a716-44665544000", "x"] {
+ let c =
ChannelBuilder::parse(&format!("sc://localhost/;session_id={id}")).unwrap();
+ let e = c.session_id().unwrap_err();
+ assert_eq!(e.error_class, "INVALID_SESSION_UUID_ID", "{id}");
+ }
}
}
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index aedbec9e..7d940384 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -293,7 +293,7 @@ impl SparkConnectClient {
// client can be constructed without a live server (e.g. in unit tests
that only
// build request protos).
let endpoint = builder.endpoint();
- let channel = if builder.use_ssl() {
+ let channel = if builder.secure() {
// TLS connection with native root certificates
// `with_native_roots()` is required: tonic >=0.11 does NOT load a
trust store
// from the `tls-native-roots` feature alone, so without this the
TLS handshake
@@ -330,24 +330,9 @@ impl SparkConnectClient {
// excluded from `metadata()`, so attach it explicitly here; this then
propagates
// to every request via `_attach_metadata` and the reattach stream.
if let Some(tok) = builder.token() {
- // Never send the bearer token in cleartext. The scheme above is
chosen from
- // `use_ssl()` alone, so without this guard
`sc://host/;token=SECRET` (no
- // `use_ssl`) would ship `Authorization: Bearer SECRET` over plain
HTTP. The
- // reference client cannot reach that state: `toChannel` sends a
token only
- // over `ssl_channel_credentials()` or, for `localhost`, the
loopback
- // `local_channel_credentials()`, and grpcio refuses call
credentials on an
- // insecure channel outright. Mirror that by requiring TLS for a
token unless
- // the endpoint is loopback; fail loudly rather than downgrade
silently.
- if !builder.use_ssl() && !builder.is_loopback() {
- // A misconfigured connection string is a value error
(surfaces as
- // `PySparkValueError`), not a generic runtime failure.
- return Err(SparkError::value_msg(format!(
- "Refusing to send the authentication token to '{}' over an
insecure \
- connection. Set 'use_ssl=true' in the connection string
to enable TLS \
- (a token is only allowed without TLS when connecting to
localhost).",
- builder.host()
- )));
- }
+ // The token is sent over TLS, or over plaintext only on a
loopback host: the
+ // scheme above is `builder.secure()`, which is TLS for a token on
any other
+ // host, as the reference client does.
metadata.push((AUTHORIZATION_HEADER.to_string(), format!("Bearer
{}", tok)));
}
@@ -355,7 +340,7 @@ impl SparkConnectClient {
channel,
stub,
session_id,
- user_id: builder.user_id().map(String::from),
+ user_id:
builder.user_id().map(String::from).or_else(default_user_id),
metadata,
user_agent,
user_context_extensions: Default::default(),
@@ -1552,6 +1537,21 @@ pub struct ReattachableResponseStream {
released: bool,
}
+/// The user ID of a connection string that does not set one: `SPARK_USER`,
else `USER`, as
+/// `SparkConnectClient.__init__` in PySpark takes it.
+fn default_user_id() -> Option<String> {
+ user_id_from_env(std::env::var("SPARK_USER"), std::env::var("USER"))
+}
+
+/// The user ID from the environment, as `os.getenv("SPARK_USER",
os.getenv("USER"))`: `USER`
+/// is only read when `SPARK_USER` is unset. An empty value is no user ID.
+fn user_id_from_env(
+ spark_user: std::result::Result<String, std::env::VarError>,
+ user: std::result::Result<String, std::env::VarError>,
+) -> Option<String> {
+ spark_user.or(user).ok().filter(|u| !u.is_empty())
+}
+
impl SparkConnectClient {
/// The gRPC metadata of every request about an execution.
fn execution_metadata(&self) -> MetadataMap {
@@ -2072,20 +2072,98 @@ mod tests {
);
}
- #[tokio::test]
- async fn test_token_without_ssl_to_remote_host_is_rejected() {
- // A token over a non-loopback host without `use_ssl=true` would ship
the bearer
- // credential in cleartext; the client must refuse rather than
downgrade silently.
- let builder =
ChannelBuilder::parse("sc://example.com/;token=SECRET").unwrap();
- match SparkConnectClient::connect(&builder).await {
- Ok(_) => panic!("expected a token over cleartext to a remote host
to be rejected"),
- Err(e) => assert!(
- e.to_string().contains("use_ssl=true"),
- "expected the error to point at use_ssl=true, got: {e}"
- ),
+ /// Connects a client on a runtime of its own, for the tests that set the
environment.
+ /// The channel is lazy, so no server is dialed.
+ fn connect_blocking(url: &str) -> SparkConnectClient {
+ let builder = ChannelBuilder::parse(url).unwrap();
+ let runtime = tokio::runtime::Builder::new_current_thread()
+ .enable_all()
+ .build()
+ .unwrap();
+ runtime
+ .block_on(SparkConnectClient::connect(&builder))
+ .unwrap_or_else(|e| panic!("{url} should connect, got: {e}"))
+ }
+
+ fn has_bearer(client: &SparkConnectClient, token: &str) -> bool {
+ let bearer = format!("Bearer {token}");
+ client
+ .metadata
+ .iter()
+ .any(|(k, v)| k == "authorization" && *v == bearer)
+ }
+
+ #[test]
+ fn test_token_without_ssl_to_remote_host_is_sent_over_tls() {
+ // A token over a remote host is not refused: its channel is TLS,
which the
+ // reference client also uses for a token on any host but localhost.
+ let _env =
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+ for url in [
+ "sc://example.com/;token=SECRET",
+ "sc://10.0.0.1:443/;token=SECRET",
+ ] {
+ assert!(ChannelBuilder::parse(url).unwrap().secure(), "{url}");
+ let client = connect_blocking(url);
+ assert!(
+ has_bearer(&client, "SECRET"),
+ "{url}: {:?}",
+ client.metadata
+ );
}
}
+ #[test]
+ fn test_token_of_the_environment_to_remote_host_is_sent_over_tls() {
+ let env =
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", Some("ENV"))]);
+ assert!(ChannelBuilder::parse("sc://example.com").unwrap().secure());
+ let client = connect_blocking("sc://example.com");
+ assert!(has_bearer(&client, "ENV"), "{:?}", client.metadata);
+ drop(env);
+ // Without a token, nothing is attached.
+ let _none =
crate::channel::set_env(&[("SPARK_CONNECT_AUTHENTICATE_TOKEN", None)]);
+ let client = connect_blocking("sc://example.com");
+ assert!(!has_bearer(&client, "ENV"), "{:?}", client.metadata);
+ }
+
+ #[test]
+ fn test_user_id_from_the_environment_as_pyspark() {
+ let some = |s: &str| Ok::<String, std::env::VarError>(s.to_string());
+ let unset = || Err::<String,
std::env::VarError>(std::env::VarError::NotPresent);
+ assert_eq!(
+ user_id_from_env(some("alice"), some("bob")),
+ Some("alice".into())
+ );
+ assert_eq!(user_id_from_env(unset(), some("bob")), Some("bob".into()));
+ // A `SPARK_USER` that is set to an empty string is the user ID, so
`USER` is not read.
+ assert_eq!(user_id_from_env(some(""), some("bob")), None);
+ assert_eq!(user_id_from_env(unset(), some("")), None);
+ assert_eq!(user_id_from_env(unset(), unset()), None);
+ }
+
+ #[test]
+ fn test_user_id_of_the_connection_then_the_environment() {
+ // The connection string's user_id, else SPARK_USER, else USER, as
PySpark does.
+ let user_id = |url: &str, spark_user: Option<&str>, user:
Option<&str>| {
+ let _env = crate::channel::set_env(&[("SPARK_USER", spark_user),
("USER", user)]);
+ let client = connect_blocking(url);
+ let id = client.user_id().map(String::from);
+ id
+ };
+ assert_eq!(
+ user_id("sc://localhost/;user_id=url", Some("alice"),
Some("bob")).as_deref(),
+ Some("url")
+ );
+ assert_eq!(
+ user_id("sc://localhost", Some("alice"), Some("bob")).as_deref(),
+ Some("alice")
+ );
+ assert_eq!(
+ user_id("sc://localhost", None, Some("bob")).as_deref(),
+ Some("bob")
+ );
+ assert_eq!(user_id("sc://localhost", None, None), None);
+ }
+
#[tokio::test]
async fn test_token_without_ssl_to_localhost_is_allowed() {
// Local development mirrors upstream's `local_channel_credentials()`
exception:
diff --git a/crates/spark-connect-core/tests/user_context.rs
b/crates/spark-connect-core/tests/user_context.rs
index 6dce9b7c..09c9831d 100644
--- a/crates/spark-connect-core/tests/user_context.rs
+++ b/crates/spark-connect-core/tests/user_context.rs
@@ -425,21 +425,22 @@ async fn
typed_requests_take_a_missing_user_id_from_the_connection() {
}
#[tokio::test]
-async fn typed_requests_keep_their_user_context_without_a_configured_user_id()
{
+async fn typed_requests_keep_their_user_context_with_an_empty_user_id() {
let explicit = UserContext {
user_id: "explicit".into(),
user_name: "name".into(),
extensions: vec![extension()],
};
for (given, expected) in [
- // The message is still sent, as for every RPC that the client builds
itself.
+ // The message is still sent, as for every RPC that the client builds
itself. An
+ // empty `user_id` in the connection string is no user, whatever
SPARK_USER or USER is.
(None, UserContext::default()),
(Some(UserContext::default()), UserContext::default()),
(Some(explicit.clone()), explicit),
] {
let fixture = Fixture::start([Execution::Complete,
Execution::Complete]).await;
- let client = fixture.client(None).await;
- assert_eq!(client.user_id(), None);
+ let client = fixture.client(Some("")).await;
+ assert_eq!(client.user_id(), Some(""));
for (method, context) in send_typed_requests(&fixture, &client,
&given).await {
assert_eq!(context.as_ref(), Some(&expected), "{method} for
{given:?}");
}
diff --git a/docs/configuration.md b/docs/configuration.md
index 58e25c32..486c8b1d 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -8,7 +8,7 @@ Connection strings follow the format
`sc://host:port/;param=value`. Parameters a
| Parameter | Purpose | Example |
| --- | --- | --- |
-| `token` | Authentication token | `sc://host:15002/;token=abc123` |
+| `token` | Authentication token, sent over TLS to any host but localhost |
`sc://host:15002/;token=abc123` |
| `user_id` | User identity | `sc://host:15002/;user_id=alice` |
| `session_id` | Session identifier | `sc://host:15002/;session_id=sess_xyz` |
| `use_ssl` | Enable TLS | `sc://host:15002/;use_ssl=true` |
@@ -71,12 +71,12 @@ let spark = SparkSessionBuilder::default()
.get_or_create()?;
```
-Replace `spark.example.com`, the port, and authentication parameters as
needed. If TLS is required:
+Replace `spark.example.com`, the port, and authentication parameters as
needed. As in PySpark, a token implies TLS, so this connects over TLS. A token
of the `SPARK_CONNECT_AUTHENTICATE_TOKEN` environment variable does the same.
Only to `localhost`, `127.0.0.1` and `[::1]` is a token sent without TLS, for
local development. Without a token, `use_ssl=true` turns TLS on:
```rust
use spark_connect::SparkSessionBuilder;
let spark = SparkSessionBuilder::default()
- .remote("sc://spark.example.com:15002/;use_ssl=true;token=YOUR_TOKEN")
+ .remote("sc://spark.example.com:15002/;use_ssl=true")
.get_or_create()?;
```
diff --git a/python/tests/test_arrow_handoff.py
b/python/tests/test_arrow_handoff.py
index 60563cd1..cfe35c1f 100644
--- a/python/tests/test_arrow_handoff.py
+++ b/python/tests/test_arrow_handoff.py
@@ -717,6 +717,8 @@ def
test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch, u
# Exercise ChannelBuilder's default prefix when no URL or environment
value is set.
monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+ # Without a user_id in the URL, the user ID is SPARK_USER, as PySpark's.
+ monkeypatch.setenv("SPARK_USER", "env-user")
table = pa.table({"id": range(10_000)})
schema = T.StructType([T.StructField("id", T.LongType())])
with _arrow_input_service(protocol, user_id=user_id,
user_agent=user_agent, **{
@@ -731,7 +733,7 @@ def
test_cached_arrow_upload_attribution_on_every_frame(protocol, monkeypatch, u
_assert_artifact_contexts(
state["upload_contexts"] + state["status_contexts"],
session_id=spark.session_id,
- user_id=user_id or "",
+ user_id=user_id or "env-user",
user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
)
@@ -745,6 +747,8 @@ def test_file_upload_attribution_on_every_frame(
protocol, monkeypatch, tmp_path, user_id, user_agent, upload
):
monkeypatch.delenv("SPARK_CONNECT_USER_AGENT", raising=False)
+ # Without a user_id in the URL, the user ID is SPARK_USER, as PySpark's.
+ monkeypatch.setenv("SPARK_USER", "env-user")
small, large = tmp_path / "small.txt", tmp_path / "large.txt"
small.write_bytes(b"small artifact")
chunk_size = 32 * 1024 # Mirrors CHUNK_SIZE in
crates/spark-connect-core/src/artifact.rs.
@@ -776,7 +780,7 @@ def test_file_upload_attribution_on_every_frame(
_assert_artifact_contexts(
state["upload_contexts"],
session_id=spark.session_id,
- user_id=user_id or "",
+ user_id=user_id or "env-user",
user_agent=user_agent or "_SPARK_CONNECT_PYTHON",
)
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index cdeca982..92b41e2c 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -1131,6 +1131,29 @@ def
test_repartition_by_range_without_a_count_sends_none_as_pyspark(mock_connect
assert sent(df.repartitionByRange(2, "id")) == [2]
+def
test_session_id_of_any_uuid_version_and_the_user_id_of_the_requests(monkeypatch,
mock_connect):
+ # A session_id is any UUID, as uuid.UUID(session_id, version=4) takes it,
and is sent as
+ # given. The user ID of each request is SPARK_USER, else USER, as PySpark
sets it.
+ from pyspark.sql import SparkSession
+
+ _, service = mock_connect
+ session_id = "6ba7b810-9dad-11d1-80b4-00c04fd430c8" # a version 1 UUID
+ for name, environment in (("SPARK_USER", "hook-user"), ("USER",
"hook-os-user")):
+ monkeypatch.delenv("SPARK_USER", raising=False)
+ monkeypatch.delenv("USER", raising=False)
+ monkeypatch.setenv(name, environment)
+ other =
SparkSession.builder.remote(f"{service.remote}/;session_id={session_id}").create()
+ try:
+ assert other.session_id == session_id
+ other.range(1).collect()
+ # ExecutePlanRequest.session_id=1, user_context=2;
UserContext.user_id=1.
+ request = parse_message(service.execute_requests[-1])
+ assert request[1][0].decode() == session_id
+ assert parse_message(request[2][0])[1][0].decode() == environment
+ finally:
+ other.stop()
+
+
def test_channel_builder_uses_endpoint():
# channelBuilder reconstructs an sc:// URL from the builder's host/port (+
params)
# for the native transport, returning a configured Builder (no longer
raises).
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]