This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 7e07fef [SPARK-60989][CONNECT] Populate user context on typed requests
7e07fef is described below
commit 7e07fef9c26acd03e94fde9cc4220daed4c94312
Author: WindsorWZZ <[email protected]>
AuthorDate: Sat Nov 7 09:02:18 2026 +0900
[SPARK-60989][CONNECT] Populate user context on typed requests
### What changes were proposed in this pull request?
The typed RPCs of `SparkConnectClient` that send a request built by the
caller now take a missing `user_context.user_id` from the connection's
`user_id`, as PySpark sets it on the requests it builds: ExecutePlan (also
before the reattachable execution's iterator is built, so that its
ReattachExecute and ReleaseExecute requests carry the same context),
AnalyzePlan, ReattachExecute, ReleaseExecute, Config, Interrupt and
FetchErrorDetails. A missing `user_context` message is added. A use [...]
### Why are the changes needed?
[SPARK-60989](https://issues.apache.org/jira/browse/SPARK-60989): the
server keys a session by the user ID in each request. The client already sends
the user ID in the requests it builds itself, such as configs, interrupts and
artifact uploads, but the ExecutePlan and AnalyzePlan requests of the DataFrame
API carried an empty one. So with `user_id` in the connection string, queries
ran in another server session than configs and artifacts. Against Spark 4.3.0:
```python
spark =
SparkSession.builder.remote("sc://localhost:15002/;user_id=u").getOrCreate()
spark.conf.set("spark.sql.shuffle.partitions", "7")
spark.sql("SET spark.sql.shuffle.partitions").collect()[0][1]
# '200' before this change; '7' with it, as with PySpark 4.3.0
```
### Does this PR introduce _any_ user-facing change?
Yes. With `user_id` in the connection string, queries run in the same
server session as configs and artifact uploads, as in PySpark.
### How was this patch tested?
- `crates/spark-connect-core/tests/user_context.rs`, against a loopback
gRPC service that records the bytes of every request (as
`tests/reattach_cleanup.rs` does), with and without a configured user ID: every
typed RPC fills a missing ID and keeps an explicit context; the ReattachExecute
and ReleaseExecute requests of a reattachable execution carry its context after
a reattach, after completion and after an early drop of the stream, waiting for
the background releases rather than coun [...]
- A live test in `python/tests/test_dropin_e2e.py`: with `user_id` in the
connection string, a config set with `spark.conf.set` applies to the queries,
and back. It fails on the current master and passes with official PySpark 4.3.0.
- Against the current master and a local Spark 4.3.0 server: the lib and
every integration test of `apache-spark-connect` and
`apache-spark-connect-core`, e2e included (819 passed), the build of all tests
with `wasm-udf`, the offline Python suites (1,810 passed, 8 skipped), the live
drop-in and Arrow suites (110 passed) and the live cases (10 passed). `cargo
fmt --all --check` passes.
Changes on top of the original commits: the merge with master (keeping
master's artifact stamping and execution cleanup), filling the context on the
remaining typed RPCs, and moving the tests to a loopback service, which drops
the in-memory transport and its dev-dependencies (`hyper-util`, `tower`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6); the rework on the current master by
Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #140 from WindsorWZZ/windsorwzz/typed-user-context.
Lead-authored-by: WindsorWZZ <[email protected]>
Co-authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/spark-connect-core/src/client.rs | 45 +-
crates/spark-connect-core/tests/user_context.rs | 535 ++++++++++++++++++++++++
python/tests/test_dropin_e2e.py | 18 +
3 files changed, 590 insertions(+), 8 deletions(-)
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 7712244..786192a 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -325,8 +325,9 @@ impl SparkConnectClient {
/// Mirrors
`pyspark.sql.connect.client.core.SparkConnectClient.execute_plan`.
pub async fn execute_plan(
&self,
- request: ExecutePlanRequest,
+ mut request: ExecutePlanRequest,
) -> Result<Streaming<ExecutePlanResponse>> {
+ self.fill_user_context(&mut request.user_context);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
match self.stub.clone().execute_plan(req).await {
@@ -380,8 +381,9 @@ impl SparkConnectClient {
/// Mirrors the `ReattachExecute` RPC used by the reattachable execute
path.
pub async fn reattach_execute(
&self,
- request: ReattachExecuteRequest,
+ mut request: ReattachExecuteRequest,
) -> Result<Streaming<ExecutePlanResponse>> {
+ self.fill_user_context(&mut request.user_context);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
match self.stub.clone().reattach_execute(req).await {
@@ -395,8 +397,9 @@ impl SparkConnectClient {
/// Mirrors the `ReleaseExecute` RPC used by the reattachable execute path.
pub async fn release_execute(
&self,
- request: ReleaseExecuteRequest,
+ mut request: ReleaseExecuteRequest,
) -> Result<ReleaseExecuteResponse> {
+ self.fill_user_context(&mut request.user_context);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
match self.stub.clone().release_execute(req).await {
@@ -432,7 +435,11 @@ impl SparkConnectClient {
/// Analyze a plan and return metadata about it.
///
/// Mirrors
`pyspark.sql.connect.client.core.SparkConnectClient.analyze_plan`.
- pub async fn analyze_plan(&self, request: AnalyzePlanRequest) ->
Result<AnalyzePlanResponse> {
+ pub async fn analyze_plan(
+ &self,
+ mut request: AnalyzePlanRequest,
+ ) -> Result<AnalyzePlanResponse> {
+ self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
let mut req = with_deadline(
Request::new(request.clone()),
@@ -480,8 +487,11 @@ impl SparkConnectClient {
/// Mirrors the reference reattachable execute path.
pub async fn execute_plan_reattachable(
&self,
- request: ExecutePlanRequest,
+ mut request: ExecutePlanRequest,
) -> Result<ReattachableResponseStream> {
+ // Before the iterator: its ReattachExecute and ReleaseExecute
requests take the
+ // user context of this request.
+ self.fill_user_context(&mut request.user_context);
let iter = ExecutePlanResponseReattachableIterator::new(request);
let start = iter.request().clone();
let deadline = self.rpc_deadlines.reattachable_execute_plan;
@@ -526,8 +536,9 @@ impl SparkConnectClient {
/// server-side exception message (e.g. wrapped worker errors).
pub async fn fetch_error_details(
&self,
- request: FetchErrorDetailsRequest,
+ mut request: FetchErrorDetailsRequest,
) -> Result<FetchErrorDetailsResponse> {
+ self.fill_user_context(&mut request.user_context);
let mut req = with_deadline(
Request::new(request),
self.rpc_deadlines.fetch_error_details,
@@ -621,7 +632,8 @@ impl SparkConnectClient {
/// Update or fetch configurations.
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.config`.
- pub async fn config(&self, request: ConfigRequest) ->
Result<ConfigResponse> {
+ pub async fn config(&self, mut request: ConfigRequest) ->
Result<ConfigResponse> {
+ self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
let mut req = with_deadline(Request::new(request.clone()),
self.rpc_deadlines.config);
self._attach_metadata(&mut req);
@@ -634,7 +646,8 @@ impl SparkConnectClient {
/// Interrupt running operations on this session.
///
/// Mirrors `pyspark.sql.connect.client.core.SparkConnectClient.interrupt`.
- pub async fn interrupt(&self, request: InterruptRequest) ->
Result<InterruptResponse> {
+ pub async fn interrupt(&self, mut request: InterruptRequest) ->
Result<InterruptResponse> {
+ self.fill_user_context(&mut request.user_context);
self.with_retry(|| {
let mut req =
with_deadline(Request::new(request.clone()),
self.rpc_deadlines.interrupt);
@@ -1068,6 +1081,22 @@ impl SparkConnectClient {
Ok(hashes)
}
+ /// The identity of a typed request that the caller built: adds the user
context when it
+ /// is missing, and takes a missing user ID from the connection (`user_id`
of the
+ /// channel), as the reference client sets `user_context.user_id` on the
requests it
+ /// builds. The server keys a session by the user ID of each request, so
without it the
+ /// request would reach another session than the RPCs this client builds
itself. An
+ /// explicit user ID, the user name and the extensions are kept; raw
requests are sent
+ /// as given.
+ fn fill_user_context(&self, context: &mut Option<UserContext>) {
+ let context = context.get_or_insert_with(UserContext::default);
+ if context.user_id.is_empty() {
+ if let Some(user_id) = &self.user_id {
+ context.user_id.clone_from(user_id);
+ }
+ }
+ }
+
/// PySpark and Scala ArtifactManager both send a UserContext even when no
/// user ID is configured.
fn artifact_user_context(&self) -> UserContext {
diff --git a/crates/spark-connect-core/tests/user_context.rs
b/crates/spark-connect-core/tests/user_context.rs
new file mode 100644
index 0000000..6dce9b7
--- /dev/null
+++ b/crates/spark-connect-core/tests/user_context.rs
@@ -0,0 +1,535 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The user context of typed requests on the wire, against a loopback gRPC
service that
+//! records the bytes of every request; no Spark server required.
+
+use std::collections::VecDeque;
+use std::convert::Infallible;
+use std::pin::Pin;
+use std::sync::{Arc, Mutex};
+use std::task::{Context, Poll};
+use std::time::Duration;
+
+use futures::{stream, Stream, StreamExt};
+use prost::Message;
+use spark_connect_core::bytes_codec::BytesCodec;
+use spark_connect_core::{ChannelBuilder, RetryPolicy, SparkConnectClient};
+use spark_connect_proto::execute_plan_response::{ResponseType, ResultComplete};
+use spark_connect_proto::release_execute_request::Release;
+use spark_connect_proto::{
+ AnalyzePlanRequest, ConfigRequest, ExecutePlanRequest, ExecutePlanResponse,
+ FetchErrorDetailsRequest, InterruptRequest, ReattachExecuteRequest,
ReleaseExecuteRequest,
+ UserContext,
+};
+use tokio::net::TcpListener;
+use tokio::sync::Notify;
+use tonic::body::Body;
+use tonic::codegen::{http, BoxFuture, Service};
+use tonic::server::{NamedService, ServerStreamingService, UnaryService};
+use tonic::transport::Server;
+use tonic::{Request, Response, Status};
+
+const WAIT: Duration = Duration::from_secs(5);
+const SERVICE: &str = "/spark.connect.SparkConnectService/";
+type Frames = Pin<Box<dyn Stream<Item = Result<Vec<u8>, Status>> + Send>>;
+
+/// What an ExecutePlan call streams. A ReattachExecute call streams the rest
of the result.
+#[derive(Clone, Copy)]
+enum Execution {
+ /// The whole result.
+ Complete,
+ /// A first response, then nothing until the client goes away.
+ Open,
+ /// A first response, then the stream drops, so that the client reattaches.
+ Disconnect,
+}
+
+struct State {
+ executions: Mutex<VecDeque<Execution>>,
+ /// The method and the bytes of every request, in the order they arrived.
+ requests: Mutex<Vec<(String, Vec<u8>)>>,
+ changed: Notify,
+}
+
+impl State {
+ fn record(&self, method: &str, request: Vec<u8>) {
+ self.requests
+ .lock()
+ .unwrap()
+ .push((method.to_string(), request));
+ self.changed.notify_waiters();
+ }
+
+ fn requests_of(&self, method: &str) -> Vec<Vec<u8>> {
+ self.requests
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|(m, _)| m == method)
+ .map(|(_, request)| request.clone())
+ .collect()
+ }
+}
+
+struct Fixture {
+ state: Arc<State>,
+ server: tokio::task::JoinHandle<()>,
+ address: std::net::SocketAddr,
+}
+
+impl Fixture {
+ async fn start(executions: impl IntoIterator<Item = Execution>) -> Self {
+ let state = Arc::new(State {
+ executions: Mutex::new(executions.into_iter().collect()),
+ requests: Mutex::new(Vec::new()),
+ changed: Notify::new(),
+ });
+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let incoming = stream::unfold(listener, |listener| async {
+ Some((listener.accept().await.map(|(socket, _)| socket), listener))
+ });
+ let service = TestService(state.clone());
+ let server = tokio::spawn(async move {
+ Server::builder()
+ .add_service(service)
+ .serve_with_incoming(incoming)
+ .await
+ .unwrap();
+ });
+ Self {
+ state,
+ server,
+ address,
+ }
+ }
+
+ /// A client of the fixture, with `user_id` in its connection string when
given.
+ async fn client(&self, user_id: Option<&str>) -> SparkConnectClient {
+ let url = match user_id {
+ Some(user_id) => format!("sc://{}/;user_id={user_id}",
self.address),
+ None => format!("sc://{}", self.address),
+ };
+ let builder = ChannelBuilder::parse(&url).unwrap();
+ let mut client = tokio::time::timeout(WAIT,
SparkConnectClient::connect(&builder))
+ .await
+ .expect("connecting timed out")
+ .unwrap();
+ client.set_retry_policy(RetryPolicy {
+ max_retries: Some(2),
+ initial_backoff_ms: 1,
+ max_backoff_ms: Some(2),
+ ..RetryPolicy::default()
+ });
+ client
+ }
+
+ /// The decoded requests of `method`, once `done` holds for them.
ReleaseExecute requests
+ /// are sent in the background, so they are waited for rather than counted
at once.
+ async fn wait_for<T: Message + Default>(
+ &self,
+ method: &str,
+ done: impl Fn(&[T]) -> bool,
+ ) -> Vec<T> {
+ tokio::time::timeout(WAIT, async {
+ loop {
+ let changed = self.state.changed.notified();
+ let requests: Vec<T> = self
+ .state
+ .requests_of(method)
+ .iter()
+ .map(|bytes| T::decode(bytes.as_slice()).unwrap())
+ .collect();
+ if done(&requests) {
+ return requests;
+ }
+ changed.await;
+ }
+ })
+ .await
+ .unwrap_or_else(|_| panic!("the fixture did not receive the expected
{method} requests"))
+ }
+}
+
+impl Drop for Fixture {
+ fn drop(&mut self) {
+ self.server.abort();
+ }
+}
+
+fn response(id: &str, complete: bool) -> Vec<u8> {
+ ExecutePlanResponse {
+ response_id: id.into(),
+ response_type:
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
+ ..Default::default()
+ }
+ .encode_to_vec()
+}
+
+#[derive(Clone)]
+struct TestService(Arc<State>);
+
+impl NamedService for TestService {
+ const NAME: &'static str = "spark.connect.SparkConnectService";
+}
+
+impl Service<http::Request<Body>> for TestService {
+ type Response = http::Response<Body>;
+ type Error = Infallible;
+ type Future = BoxFuture<Self::Response, Self::Error>;
+
+ fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(),
Self::Error>> {
+ Poll::Ready(Ok(()))
+ }
+
+ fn call(&mut self, request: http::Request<Body>) -> Self::Future {
+ let method =
request.uri().path().trim_start_matches(SERVICE).to_string();
+ let state = self.0.clone();
+ Box::pin(async move {
+ let mut grpc = tonic::server::Grpc::new(BytesCodec);
+ Ok(match method.as_str() {
+ "ExecutePlan" | "ReattachExecute" => {
+ grpc.server_streaming(Streamed(state, method), request)
+ .await
+ }
+ _ => grpc.unary(Unary(state, method), request).await,
+ })
+ })
+ }
+}
+
+/// ExecutePlan and ReattachExecute.
+struct Streamed(Arc<State>, String);
+
+impl ServerStreamingService<Vec<u8>> for Streamed {
+ type Response = Vec<u8>;
+ type ResponseStream = Frames;
+ type Future = BoxFuture<Response<Frames>, Status>;
+
+ fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
+ self.0.record(&self.1, request.into_inner());
+ let execution = match self.1.as_str() {
+ "ExecutePlan" => self.0.executions.lock().unwrap().pop_front(),
+ _ => None,
+ };
+ let frames: Frames = match execution.unwrap_or(Execution::Complete) {
+ Execution::Complete =>
Box::pin(stream::iter([Ok(response("complete", true))])),
+ Execution::Open => {
+ Box::pin(stream::iter([Ok(response("first",
false))]).chain(stream::pending()))
+ }
+ Execution::Disconnect => Box::pin(stream::iter([
+ Ok(response("first", false)),
+ Err(Status::unavailable("stream disconnected")),
+ ])),
+ };
+ Box::pin(async { Ok(Response::new(frames)) })
+ }
+}
+
+/// The unary RPCs, each answered with its empty response message.
+struct Unary(Arc<State>, String);
+
+impl UnaryService<Vec<u8>> for Unary {
+ type Response = Vec<u8>;
+ type Future = BoxFuture<Response<Vec<u8>>, Status>;
+
+ fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
+ self.0.record(&self.1, request.into_inner());
+ Box::pin(async { Ok(Response::new(Vec::new())) })
+ }
+}
+
+fn extension() -> prost_types::Any {
+ prost_types::Any {
+ type_url: "example.org/context".into(),
+ value: vec![1, 2, 3],
+ }
+}
+
+fn is_release_all(request: &ReleaseExecuteRequest) -> bool {
+ matches!(request.release, Some(Release::ReleaseAll(_)))
+}
+
+/// Sends every typed request of the client with `context` as its user
context, and returns
+/// the user contexts that the server received, by method.
+async fn send_typed_requests(
+ fixture: &Fixture,
+ client: &SparkConnectClient,
+ context: &Option<UserContext>,
+) -> Vec<(&'static str, Option<UserContext>)> {
+ let execute = ExecutePlanRequest {
+ session_id: client.session_id().to_string(),
+ user_context: context.clone(),
+ ..Default::default()
+ };
+ let mut stream = client.execute_plan(execute.clone()).await.unwrap();
+ while stream.message().await.unwrap().is_some() {}
+ let mut stream = client.execute_plan_reattachable(execute).await.unwrap();
+ while stream.message().await.unwrap().is_some() {}
+ client
+ .analyze_plan(AnalyzePlanRequest {
+ user_context: context.clone(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+ client
+ .config(ConfigRequest {
+ user_context: context.clone(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+ client
+ .interrupt(InterruptRequest {
+ user_context: context.clone(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+ client
+ .fetch_error_details(FetchErrorDetailsRequest {
+ user_context: context.clone(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+ let mut stream = client
+ .reattach_execute(ReattachExecuteRequest {
+ user_context: context.clone(),
+ operation_id: "typed-operation".into(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+ while stream.message().await.unwrap().is_some() {}
+ client
+ .release_execute(ReleaseExecuteRequest {
+ user_context: context.clone(),
+ operation_id: "typed-operation".into(),
+ ..Default::default()
+ })
+ .await
+ .unwrap();
+
+ let mut received = Vec::new();
+ for request in fixture
+ .wait_for::<ExecutePlanRequest>("ExecutePlan", |r| r.len() == 2)
+ .await
+ {
+ received.push(("ExecutePlan", request.user_context));
+ }
+ for request in fixture
+ .wait_for::<AnalyzePlanRequest>("AnalyzePlan", |r| r.len() == 1)
+ .await
+ {
+ received.push(("AnalyzePlan", request.user_context));
+ }
+ for request in fixture
+ .wait_for::<ConfigRequest>("Config", |r| r.len() == 1)
+ .await
+ {
+ received.push(("Config", request.user_context));
+ }
+ for request in fixture
+ .wait_for::<InterruptRequest>("Interrupt", |r| r.len() == 1)
+ .await
+ {
+ received.push(("Interrupt", request.user_context));
+ }
+ for request in fixture
+ .wait_for::<FetchErrorDetailsRequest>("FetchErrorDetails", |r| r.len()
== 1)
+ .await
+ {
+ received.push(("FetchErrorDetails", request.user_context));
+ }
+ for request in fixture
+ .wait_for::<ReattachExecuteRequest>("ReattachExecute", |r| r.len() ==
1)
+ .await
+ {
+ received.push(("ReattachExecute", request.user_context));
+ }
+ // The typed release, and the release of the completed reattachable
execution, which the
+ // client sends in the background.
+ let releases = fixture
+ .wait_for::<ReleaseExecuteRequest>("ReleaseExecute", |r| {
+ r.iter().any(|r| r.operation_id == "typed-operation")
+ && r.iter()
+ .any(|r| r.operation_id != "typed-operation" &&
is_release_all(r))
+ })
+ .await;
+ for request in releases {
+ received.push(("ReleaseExecute", request.user_context));
+ }
+ received
+}
+
+#[tokio::test]
+async fn typed_requests_take_a_missing_user_id_from_the_connection() {
+ let partial = UserContext {
+ user_name: "name".into(),
+ extensions: vec![extension()],
+ ..Default::default()
+ };
+ let explicit = UserContext {
+ user_id: "explicit".into(),
+ user_name: "name".into(),
+ extensions: vec![extension()],
+ };
+ for (given, expected) in [
+ (
+ None,
+ UserContext {
+ user_id: "configured".into(),
+ ..Default::default()
+ },
+ ),
+ (
+ Some(UserContext::default()),
+ UserContext {
+ user_id: "configured".into(),
+ ..Default::default()
+ },
+ ),
+ (
+ Some(partial.clone()),
+ UserContext {
+ user_id: "configured".into(),
+ ..partial
+ },
+ ),
+ // An explicit user ID wins over the connection's.
+ (Some(explicit.clone()), explicit),
+ ] {
+ let fixture = Fixture::start([Execution::Complete,
Execution::Complete]).await;
+ let client = fixture.client(Some("configured")).await;
+ for (method, context) in send_typed_requests(&fixture, &client,
&given).await {
+ assert_eq!(context.as_ref(), Some(&expected), "{method} for
{given:?}");
+ }
+ }
+}
+
+#[tokio::test]
+async fn typed_requests_keep_their_user_context_without_a_configured_user_id()
{
+ let explicit = UserContext {
+ user_id: "explicit".into(),
+ user_name: "name".into(),
+ extensions: vec![extension()],
+ };
+ for (given, expected) in [
+ // The message is still sent, as for every RPC that the client builds
itself.
+ (None, UserContext::default()),
+ (Some(UserContext::default()), UserContext::default()),
+ (Some(explicit.clone()), explicit),
+ ] {
+ let fixture = Fixture::start([Execution::Complete,
Execution::Complete]).await;
+ let client = fixture.client(None).await;
+ assert_eq!(client.user_id(), None);
+ for (method, context) in send_typed_requests(&fixture, &client,
&given).await {
+ assert_eq!(context.as_ref(), Some(&expected), "{method} for
{given:?}");
+ }
+ }
+}
+
+/// The ReattachExecute and ReleaseExecute requests of a reattachable
execution take the
+/// user context of its ExecutePlan request, including the user ID taken from
the connection.
+#[tokio::test]
+async fn reattach_and_release_take_the_user_context_of_the_execution() {
+ let configured = Some(UserContext {
+ user_id: "configured".into(),
+ ..Default::default()
+ });
+
+ // A stream that drops after its first response: the client releases that
response,
+ // reattaches, and releases the execution once its result is complete.
+ let fixture = Fixture::start([Execution::Disconnect]).await;
+ let client = fixture.client(Some("configured")).await;
+ let mut stream = client
+ .execute_plan_reattachable(ExecutePlanRequest::default())
+ .await
+ .unwrap();
+ let mut ids = Vec::new();
+ while let Some(response) = stream.message().await.unwrap() {
+ ids.push(response.response_id);
+ }
+ assert_eq!(ids, ["first", "complete"]);
+ let execute = fixture
+ .wait_for::<ExecutePlanRequest>("ExecutePlan", |r| r.len() == 1)
+ .await;
+ assert_eq!(execute[0].user_context, configured);
+ let reattaches = fixture
+ .wait_for::<ReattachExecuteRequest>("ReattachExecute", |r| r.len() ==
1)
+ .await;
+ assert_eq!(reattaches[0].user_context, configured);
+ assert_eq!(reattaches[0].last_response_id.as_deref(), Some("first"));
+ let releases = fixture
+ .wait_for::<ReleaseExecuteRequest>("ReleaseExecute", |r| {
+ r.iter().any(is_release_all) && r.iter().any(|r|
!is_release_all(r))
+ })
+ .await;
+ for release in &releases {
+ assert_eq!(release.user_context, configured);
+ assert_eq!(
+ Some(&release.operation_id),
+ execute[0].operation_id.as_ref()
+ );
+ }
+ drop(stream);
+
+ // A stream dropped before its result is complete releases its execution.
+ let fixture = Fixture::start([Execution::Open]).await;
+ let client = fixture.client(Some("configured")).await;
+ let mut stream = client
+ .execute_plan_reattachable(ExecutePlanRequest::default())
+ .await
+ .unwrap();
+ assert_eq!(
+ stream.message().await.unwrap().unwrap().response_id,
+ "first"
+ );
+ drop(stream);
+ let releases = fixture
+ .wait_for::<ReleaseExecuteRequest>("ReleaseExecute", |r|
r.iter().any(is_release_all))
+ .await;
+ for release in &releases {
+ assert_eq!(release.user_context, configured);
+ }
+}
+
+#[tokio::test]
+async fn raw_requests_are_sent_byte_exact() {
+ let fixture = Fixture::start([Execution::Complete]).await;
+ let client = fixture.client(Some("configured")).await;
+ // Without a user context, and with an unknown field (2047, varint 1) that
must survive,
+ // as the request is not decoded and encoded again.
+ let unknown_field = [0xf8, 0x7f, 0x01];
+ let mut execute = ExecutePlanRequest::default().encode_to_vec();
+ let mut analyze = AnalyzePlanRequest::default().encode_to_vec();
+ let mut reattach = ReattachExecuteRequest::default().encode_to_vec();
+ for request in [&mut execute, &mut analyze, &mut reattach] {
+ request.extend_from_slice(&unknown_field);
+ }
+ let mut stream = client.execute_plan_raw(execute.clone()).await.unwrap();
+ while stream.message().await.unwrap().is_some() {}
+ client.analyze_plan_raw(analyze.clone()).await.unwrap();
+ let mut stream =
client.reattach_execute_raw(reattach.clone()).await.unwrap();
+ while stream.message().await.unwrap().is_some() {}
+ assert_eq!(fixture.state.requests_of("ExecutePlan"), [execute]);
+ assert_eq!(fixture.state.requests_of("AnalyzePlan"), [analyze]);
+ assert_eq!(fixture.state.requests_of("ReattachExecute"), [reattach]);
+}
diff --git a/python/tests/test_dropin_e2e.py b/python/tests/test_dropin_e2e.py
index 570261a..c42d4ec 100644
--- a/python/tests/test_dropin_e2e.py
+++ b/python/tests/test_dropin_e2e.py
@@ -104,6 +104,24 @@ def test_conf_get_with_a_none_default(session):
session.conf.get(key)
+def test_user_id_of_the_connection_string_reaches_every_request_as_pyspark():
+ # The server keys a session by the user ID in each request. As in PySpark,
the
+ # queries carry the user ID of the connection string, as the configs do,
so a config
+ # set with spark.conf.set applies to the queries and the other way around.
+ remote = os.environ.get("SPARK_REMOTE")
+ if not remote:
+ pytest.skip("SPARK_REMOTE is required for live-server tests")
+ remote += ";user_id=user-context-probe" if ";" in remote else
"/;user_id=user-context-probe"
+ spark = SparkSession.builder.remote(remote).create()
+ try:
+ spark.conf.set("spark.sql.shuffle.partitions", "7")
+ assert spark.sql("SET spark.sql.shuffle.partitions").collect()[0][1]
== "7"
+ spark.sql("SET spark.sql.shuffle.partitions=9").collect()
+ assert spark.conf.get("spark.sql.shuffle.partitions") == "9"
+ finally:
+ spark.stop()
+
+
def
test_get_or_create_returns_the_session_with_the_options_as_pyspark(session):
# As PySpark's builder: getOrCreate() returns the active session, and sets
the options
# of the builder on it; one that fails, of a static configuration, is
ignored.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]