This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark-connect-rust.git
The following commit(s) were added to refs/heads/master by this push:
new 7422948c [SPARK-61184][CONNECT] Send the server-side session ID that
the client observed, and check that of the responses, so that a session ends
when its server restarts, as PySpark does
7422948c is described below
commit 7422948c838ee0029d906251af86bd14a4c3d68b
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Mon Nov 9 08:15:12 2026 +0900
[SPARK-61184][CONNECT] Send the server-side session ID that the client
observed, and check that of the responses, so that a session ends when its
server restarts, as PySpark does
### What changes were proposed in this pull request?
[SPARK-61184](https://issues.apache.org/jira/browse/SPARK-61184): PySpark
keeps the `server_side_session_id` of the first response as
`_server_session_id`, sends it as `client_observed_server_side_session_id` on
its requests, and checks it on each response. A response with a different ID,
or an `INVALID_HANDLE.SESSION_CHANGED` error, closes the client, and its later
calls raise `NO_ACTIVE_SESSION`.
- `SparkConnectClient` keeps the observed ID in a slot shared by the clones
of a session. A new session, or a clone from `clone_session`, starts with none.
- Each typed request that has the field carries the observed ID:
ExecutePlan, AnalyzePlan, Config, Interrupt, FetchErrorDetails, CloneSession,
ReattachExecute, ReleaseExecute, AddArtifacts, ArtifactStatuses. The
ReattachExecute of a reattachable execution takes the observed ID of its
initial request, as PySpark's `reattach.py` does.
- Each typed response that has the IDs is checked: a non-empty session ID
must be this session's, and a non-empty server-side session ID that differs
from the observed one closes the client and raises `PySparkAssertionError` with
PySpark's message. The reattachable stream checks each response it reads.
- An error with the `INVALID_HANDLE.SESSION_CHANGED` error class closes the
client, in the error path of every typed RPC.
- Raw requests and responses (`*_raw`) are not changed: their bytes are
built by the caller, which sets the observed ID itself.
- `SparkConnectClient::set_session_tracking(false)` turns all of this off.
The transport-injection stub `RustConnectStub` turns it off: the requests that
it sends are of the reference client's session, which tracks its server-side
session itself, over the grpcio stub that does no tracking.
### Why are the changes needed?
After the server restarts, PySpark raises `NO_ACTIVE_SESSION` on the next
call. The Rust client sent no observed ID and checked no ID, so the calls ran
on a fresh server session. On a server restart in a session that had set
`spark.sql.shuffle.partitions=7`:
```
before (this client): conf.get -> "200"; spark.table("v4") ->
TABLE_OR_VIEW_NOT_FOUND; range(2).count() -> runs
official 4.3.0: conf.get, table and range -> SparkConnectException
NO_ACTIVE_SESSION
```
The "after" column below is from this build, against a Spark 4.3.0 Connect
server restarted in the same way (`/tmp/r2/impl/SPARK-61184/v4b.py`):
```
after restart conf -> ERR SparkConnectException NO_ACTIVE_SESSION
after restart view -> ERR SparkConnectException NO_ACTIVE_SESSION
after restart range -> ERR SparkConnectException NO_ACTIVE_SESSION
```
### Does this PR introduce _any_ user-facing change?
Yes. After a server restart, a session raises `NO_ACTIVE_SESSION` on its
calls, and its clones do too. A response with a changed server-side session ID
raises `PySparkAssertionError` ("Received incorrect server side session
identifier for request. Please create a new Spark Session to reconnect.
(...)"). Requests carry the observed ID.
Differences from PySpark:
- AddArtifacts, ArtifactStatus and ReleaseExecute carry the observed ID
here. PySpark 4.3.0 does not set it on those three requests.
- An empty server-side session ID is neither observed nor checked. PySpark
keeps an empty first ID and then raises for a later non-empty one.
- An empty session ID of a response is not checked either. The server
leaves both IDs unset in some responses, such as those of `FetchErrorDetails`,
which PySpark does not check.
- FetchErrorDetails and AddArtifacts responses are checked here. PySpark
4.3.0 does not check them.
- The streaming `execute_plan` and `reattach_execute` (used only by tests)
check no responses. The session's executions go through
`execute_plan_reattachable`, which checks each response.
### How was this patch tested?
- New `crates/spark-connect-core/tests/server_session.rs` with a loopback
gRPC server whose server-side session ID a test can change:
- `requests_carry_the_server_side_session_id_observed_in_the_responses`:
the first Config carries none, then the observed ID; ExecutePlan and the
reattach after a dropped stream carry it.
- `a_restarted_server_raises_and_closes_the_client`: a response with
another server-side session ID raises the PySpark message, closes the client
and its clone, and the following calls raise `NO_ACTIVE_SESSION`.
- `a_session_changed_error_closes_the_client`: an
`INVALID_HANDLE.SESSION_CHANGED` error closes the client, and the next call
raises `NO_ACTIVE_SESSION`.
-
`a_client_without_session_tracking_passes_the_ids_of_its_requests_through`:
with the tracking off, requests of another session, a restarted server and a
session change neither raise an assertion nor close the client, and the
requests keep the observed ID that the caller set.
- The first three tests fail without this change (the client of master
against the mock servers of this change).
- The mock servers of `deadlines.rs`, `reattach_cleanup.rs`,
`user_context.rs` and `artifact_uploads.rs` now echo the session ID of the
request, as the server does, and the connections of the first two use the
session ID of their requests. Without these, the session check fails.
- `python/tests/_mock_connect_server.py` and the two mock servers in
`python/tests/test_dropin_offline.py` echo the session ID in their analyze,
execute and clone responses. No test was added to these files.
- Against a live Spark 4.3.0 Connect server: `tests/e2e_integration.rs` (17
passed) and `test_dropin_e2e.py -k "session or conf or artifact or clone or
range"` (18 passed). The restart probe above.
- Offline `test_dropin_offline.py -k "session or conf or artifact or clone
or version"`: 60 passed.
- `cargo test -p apache-spark-connect-core --release`: all targets pass.
- The official Connect suites over our transport
(`scripts/run_official_tests.py`) for `test_parity_conf.py`,
`test_connect_error.py`, `test_parity_errors.py` and `test_connect_session.py`:
all pass. Without the stub's switch, every test of the official suites failed
in CI, because the stub checked the reference client's responses against its
own session ID. `test_parity_column.py` fails the same two
`test_column_date_time_op` cases locally with master's build too.
- The lib tests of both crates (615 passed), the build of all tests, also
with `wasm-udf`, the offline Python suites (2434 passed, 8 skipped), and the
live drop-in and Arrow suites (215 passed) and live cases (11 passed) against a
local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #385 from HyukjinKwon/SPARK-61184-server-side-session-id.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
crates/pyspark-rs/src/transport.rs | 3 +
crates/spark-connect-core/src/client.rs | 297 +++++++++---
.../spark-connect-core/tests/artifact_uploads.rs | 5 +
crates/spark-connect-core/tests/deadlines.rs | 4 +-
.../spark-connect-core/tests/reattach_cleanup.rs | 4 +-
crates/spark-connect-core/tests/server_session.rs | 505 +++++++++++++++++++++
crates/spark-connect-core/tests/user_context.rs | 93 +++-
python/tests/_mock_connect_server.py | 11 +-
python/tests/test_dropin_offline.py | 13 +-
9 files changed, 857 insertions(+), 78 deletions(-)
diff --git a/crates/pyspark-rs/src/transport.rs
b/crates/pyspark-rs/src/transport.rs
index 4386225a..7ba9eb51 100644
--- a/crates/pyspark-rs/src/transport.rs
+++ b/crates/pyspark-rs/src/transport.rs
@@ -170,6 +170,9 @@ impl RustConnectStub {
// policy - a `max_retries=0` client would otherwise hit our 15-retry
backoff and
// hang against an unreachable server instead of failing fast with
UNAVAILABLE.
client.set_retry_policy(RetryPolicy::no_retries());
+ // The requests are of the reference client's session, which tracks
its server-side
+ // session itself; its grpcio stub does no tracking.
+ client.set_session_tracking(false);
Ok(Self {
client,
cancel: Arc::new(Notify::new()),
diff --git a/crates/spark-connect-core/src/client.rs
b/crates/spark-connect-core/src/client.rs
index 7d940384..50603bc1 100644
--- a/crates/spark-connect-core/src/client.rs
+++ b/crates/spark-connect-core/src/client.rs
@@ -187,6 +187,20 @@ async fn within_deadline<T>(
}
}
+/// The server-side session ID observed by a client, from its shared slot.
+fn observed_of(slot: &std::sync::Mutex<Option<String>>) -> Option<String> {
+ slot.lock().unwrap_or_else(|e| e.into_inner()).clone()
+}
+
+/// The error of a response that does not match this session, as PySpark's
+/// `PySparkAssertionError`, which has no error class.
+fn assertion_error(message: String) -> SparkError {
+ SparkError {
+ kind: SparkErrorKind::AssertionError,
+ ..SparkError::connect_msg(message)
+ }
+}
+
/// The error class of `status` when the server does not know the operation of
a reattach, or
/// its session, which the reference reads alike: the initial ExecutePlan
never reached the
/// server, unless a response was read already.
@@ -256,6 +270,14 @@ pub struct SparkConnectClient {
/// Whether the client is closed, shared by its clones: its RPCs then raise
/// `NO_ACTIVE_SESSION`, as PySpark's `SparkConnectClient._stub` does once
it is closed.
closed: std::sync::Arc<std::sync::atomic::AtomicBool>,
+ /// The server-side session ID that the first response carried, shared by
the clones of the
+ /// client, as PySpark's `_server_session_id`. Each request sends it as
the observed ID, and
+ /// each response is checked against it.
+ server_session_id: std::sync::Arc<std::sync::Mutex<Option<String>>>,
+ /// Whether the client tracks its server-side session: the observed ID of
its requests, the
+ /// check of the IDs of its responses, and its close when the server
reports that the session
+ /// changed. See [`SparkConnectClient::set_session_tracking`].
+ session_tracking: bool,
}
/// The user context extensions of a client, as PySpark's `SparkConnectClient`
keeps them: those
@@ -348,6 +370,8 @@ impl SparkConnectClient {
rpc_deadlines: RpcDeadlines::default(),
execute_plan_hook: Default::default(),
closed: Default::default(),
+ server_session_id: Default::default(),
+ session_tracking: true,
})
}
@@ -374,6 +398,8 @@ impl SparkConnectClient {
rpc_deadlines: self.rpc_deadlines.clone(),
execute_plan_hook: std::sync::Arc::clone(&self.execute_plan_hook),
closed: Default::default(),
+ server_session_id: Default::default(),
+ session_tracking: self.session_tracking,
}
}
@@ -402,6 +428,74 @@ impl SparkConnectClient {
Ok(())
}
+ /// The server-side session ID that the responses have carried, if one has.
+ fn observed_server_session_id(&self) -> Option<String> {
+ observed_of(&self.server_session_id)
+ }
+
+ /// Sets the observed server-side session ID of a request, once a response
has set it, as
+ /// PySpark's requests do.
+ fn observe_on(&self, observed: &mut Option<String>) {
+ if !self.session_tracking {
+ return;
+ }
+ if let Some(id) = self.observed_server_session_id() {
+ *observed = Some(id);
+ }
+ }
+
+ /// Checks the IDs of a response, as PySpark's
`_verify_response_integrity`: the session ID
+ /// must be this session's. The first non-empty server-side session ID
that a response
+ /// carries is observed; a later one that differs closes the client, as
the server restarted,
+ /// and is an error. An empty ID is neither observed nor checked. PySpark
keeps an empty
+ /// first ID and then raises for a later non-empty one.
+ #[allow(clippy::result_large_err)] // SparkError is large; changing the
type would break the API
+ fn verify_response(&self, session_id: &str, server_side_session_id: &str)
-> Result<()> {
+ if !self.session_tracking {
+ return Ok(());
+ }
+ // An empty session ID is none: the server leaves both IDs unset in
some responses, such
+ // as those of FetchErrorDetails, so only another session's ID is a
mismatch.
+ if !session_id.is_empty() && session_id != self.session_id {
+ return Err(assertion_error(format!(
+ "Received incorrect session identifier for
request:{session_id} != {}",
+ self.session_id
+ )));
+ }
+ let mut observed = self
+ .server_session_id
+ .lock()
+ .unwrap_or_else(|e| e.into_inner());
+ match observed.clone() {
+ None if !server_side_session_id.is_empty() => {
+ *observed = Some(server_side_session_id.to_string());
+ }
+ None => {}
+ Some(old) if !server_side_session_id.is_empty() &&
server_side_session_id != old => {
+ drop(observed);
+ self.close();
+ return Err(assertion_error(format!(
+ "Received incorrect server side session identifier for
request. Please \
+ create a new Spark Session to reconnect.
({server_side_session_id} != {old})"
+ )));
+ }
+ Some(_) => {}
+ }
+ Ok(())
+ }
+
+ /// A gRPC failure as a [`SparkError`], which closes the client when the
server reports that
+ /// the session changed, as PySpark's `_handle_rpc_error` does.
+ fn grpc_error(&self, status: tonic::Status) -> SparkError {
+ let error = SparkError::from_grpc_status(status);
+ if self.session_tracking
+ && error.get_condition().as_deref() ==
Some("INVALID_HANDLE.SESSION_CHANGED")
+ {
+ self.close();
+ }
+ error
+ }
+
/// Set the hook of the executions of this client and of its clones, or
clear it.
pub fn set_execute_plan_hook(&self, hook: Option<ExecutePlanHook>) {
*self
@@ -548,6 +642,19 @@ impl SparkConnectClient {
self.retry_policy = policy;
}
+ /// Turn the tracking of the server-side session on (the default) or off.
With it on, the
+ /// requests carry the server-side session ID observed in the responses,
each response must
+ /// be of this session and of that server-side session, and an
+ /// `INVALID_HANDLE.SESSION_CHANGED` error closes the client, as PySpark's
+ /// `SparkConnectClient` does.
+ ///
+ /// The transport-injection stub turns it off. The requests that it sends
are of the
+ /// reference client's session, not of its own, and that client tracks its
session itself,
+ /// over the grpcio stub that does no tracking.
+ pub fn set_session_tracking(&mut self, enabled: bool) {
+ self.session_tracking = enabled;
+ }
+
/// Override the per-RPC deadlines of this client's calls, as the
`rpc_deadlines` of
/// PySpark's `SparkConnectClient`.
pub fn set_rpc_deadlines(&mut self, deadlines: RpcDeadlines) {
@@ -569,6 +676,7 @@ impl SparkConnectClient {
self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let request = self.hooked(request)?;
let mut req = Request::new(request);
self._attach_metadata(&mut req);
@@ -629,6 +737,7 @@ impl SparkConnectClient {
) -> Result<Streaming<ExecutePlanResponse>> {
self.ensure_open()?;
self.fill_user_context(&mut request.user_context);
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
match self.stub.clone().reattach_execute(req).await {
@@ -646,10 +755,15 @@ impl SparkConnectClient {
) -> Result<ReleaseExecuteResponse> {
self.ensure_open()?;
self.fill_user_context(&mut request.user_context);
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let mut req = Request::new(request);
self._attach_metadata(&mut req);
match self.stub.clone().release_execute(req).await {
- Ok(response) => Ok(response.into_inner()),
+ Ok(response) => {
+ let response = response.into_inner();
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response)
+ }
Err(status) => Err(self.error_from_status(status).await),
}
}
@@ -692,18 +806,23 @@ impl SparkConnectClient {
self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
- self.with_retry(|| {
- let deadline = self.rpc_deadlines.analyze_plan;
- let mut req = with_deadline(Request::new(request.clone()),
deadline);
- self._attach_metadata(&mut req);
- let mut stub = self.stub.clone();
- async move {
- within_deadline(deadline, stub.analyze_plan(req))
- .await
- .map(Response::into_inner)
- }
- })
- .await
+ let response = self
+ .with_retry(|| {
+ let deadline = self.rpc_deadlines.analyze_plan;
+ let mut request = request.clone();
+ self.observe_on(&mut
request.client_observed_server_side_session_id);
+ let mut req = with_deadline(Request::new(request), deadline);
+ self._attach_metadata(&mut req);
+ let mut stub = self.stub.clone();
+ async move {
+ within_deadline(deadline, stub.analyze_plan(req))
+ .await
+ .map(Response::into_inner)
+ }
+ })
+ .await?;
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response)
}
/// Run a unary RPC closure with exponential-backoff retry on transient
failures.
@@ -749,6 +868,7 @@ impl SparkConnectClient {
// user context of this request, and the request of the hook, as
PySpark's iterator
// is of the request that the hooks return.
self.fill_user_context(&mut request.user_context);
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let request = self.hooked(request)?;
let iter = ExecutePlanResponseReattachableIterator::new(request);
let start = iter.request().clone();
@@ -804,12 +924,16 @@ impl SparkConnectClient {
self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let deadline = self.rpc_deadlines.fetch_error_details;
let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
let resp = within_deadline(deadline,
self.stub.clone().fetch_error_details(req)).await;
- resp.map(Response::into_inner)
- .map_err(SparkError::from_grpc_status)
+ let response = resp
+ .map(Response::into_inner)
+ .map_err(|status| self.grpc_error(status))?;
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response)
}
/// The error for a failed RPC, as `SparkConnectClient._handle_rpc_error`
raises it. An
@@ -818,7 +942,7 @@ impl SparkConnectClient {
/// `spark.sql.connect.serverStacktrace.enabled` or
`spark.sql.pyspark.jvmStacktrace.enabled`
/// is true. As the error is raised either way, failing to read those is
not.
pub async fn error_from_status(&self, status: tonic::Status) -> SparkError
{
- let error = SparkError::from_grpc_status(status);
+ let error = self.grpc_error(status);
let Some(server) = &error.server else {
return error;
};
@@ -843,13 +967,18 @@ impl SparkConnectClient {
}),
..Default::default()
};
+ let mut request = request;
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let deadline = self.rpc_deadlines.fetch_error_details;
let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
let response = within_deadline(deadline,
self.stub.clone().fetch_error_details(req))
.await
+ .ok()?
+ .into_inner();
+ self.verify_response(&response.session_id,
&response.server_side_session_id)
.ok()?;
- Some(response.into_inner())
+ Some(response)
}
/// Whether the message of an error that the server raised shows the
server's stack
@@ -886,6 +1015,8 @@ impl SparkConnectClient {
}),
..Default::default()
};
+ let mut request = request;
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let deadline = self.rpc_deadlines.config;
let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
@@ -893,6 +1024,8 @@ impl SparkConnectClient {
.await
.ok()?
.into_inner();
+ self.verify_response(&response.session_id,
&response.server_side_session_id)
+ .ok()?;
response.pairs.into_iter().next()?.value
}
@@ -903,18 +1036,23 @@ impl SparkConnectClient {
self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
- self.with_retry(|| {
- let deadline = self.rpc_deadlines.config;
- let mut req = with_deadline(Request::new(request.clone()),
deadline);
- self._attach_metadata(&mut req);
- let mut stub = self.stub.clone();
- async move {
- within_deadline(deadline, stub.config(req))
- .await
- .map(Response::into_inner)
- }
- })
- .await
+ let response = self
+ .with_retry(|| {
+ let deadline = self.rpc_deadlines.config;
+ let mut request = request.clone();
+ self.observe_on(&mut
request.client_observed_server_side_session_id);
+ let mut req = with_deadline(Request::new(request), deadline);
+ self._attach_metadata(&mut req);
+ let mut stub = self.stub.clone();
+ async move {
+ within_deadline(deadline, stub.config(req))
+ .await
+ .map(Response::into_inner)
+ }
+ })
+ .await?;
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response)
}
/// Interrupt running operations on this session.
@@ -924,18 +1062,23 @@ impl SparkConnectClient {
self.ensure_open()?;
self.extend_user_context(&mut request.user_context);
self.fill_user_context(&mut request.user_context);
- self.with_retry(|| {
- let deadline = self.rpc_deadlines.interrupt;
- let mut req = with_deadline(Request::new(request.clone()),
deadline);
- self._attach_metadata(&mut req);
- let mut stub = self.stub.clone();
- async move {
- within_deadline(deadline, stub.interrupt(req))
- .await
- .map(Response::into_inner)
- }
- })
- .await
+ let response = self
+ .with_retry(|| {
+ let deadline = self.rpc_deadlines.interrupt;
+ let mut request = request.clone();
+ self.observe_on(&mut
request.client_observed_server_side_session_id);
+ let mut req = with_deadline(Request::new(request), deadline);
+ self._attach_metadata(&mut req);
+ let mut stub = self.stub.clone();
+ async move {
+ within_deadline(deadline, stub.interrupt(req))
+ .await
+ .map(Response::into_inner)
+ }
+ })
+ .await?;
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response)
}
/// Get configuration values from the session.
@@ -1241,7 +1384,9 @@ impl SparkConnectClient {
let response = self
.with_retry(|| {
let deadline = self.rpc_deadlines.clone_session;
- let mut req = with_deadline(Request::new(request.clone()),
deadline);
+ let mut request = request.clone();
+ self.observe_on(&mut
request.client_observed_server_side_session_id);
+ let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
let mut stub = self.stub.clone();
async move {
@@ -1251,6 +1396,7 @@ impl SparkConnectClient {
}
})
.await?;
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
if let Some(requested) = new_session_id {
if response.new_session_id != requested {
return Err(SparkError::connect_msg(format!(
@@ -1276,7 +1422,10 @@ impl SparkConnectClient {
let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
match within_deadline(deadline,
self.stub.clone().release_session(req)).await {
- Ok(_) => Ok(()),
+ Ok(response) => {
+ let response = response.into_inner();
+ self.verify_response(&response.session_id,
&response.server_side_session_id)
+ }
Err(status) => Err(self.error_from_status(status).await),
}
}
@@ -1340,9 +1489,19 @@ impl SparkConnectClient {
.cloned()
.collect();
let (blobs, names, unique) = (&blobs, &names, &unique);
- let uploaded = self
+ // The IDs of the status response, checked once the retries are done,
as its own.
+ let (status_ids, uploaded) = self
.with_retry(|| async move {
- let cached = self.artifact_status_rpc(unique).await?;
+ let status = self.artifact_status_rpc(unique).await?;
+ let status_ids = (
+ status.session_id.clone(),
+ status.server_side_session_id.clone(),
+ );
+ let cached: std::collections::HashMap<String, bool> = status
+ .statuses
+ .into_iter()
+ .map(|(name, status)| (name, status.exists))
+ .collect();
let mut seen = HashSet::new();
let mut artifacts = Vec::new();
for (blob, name) in blobs.iter().zip(names) {
@@ -1351,16 +1510,18 @@ impl SparkConnectClient {
}
}
if artifacts.is_empty() {
- return Ok(None);
+ return Ok((status_ids, None));
}
let expected: HashSet<String> =
artifacts.iter().map(|(name, _)| name.clone()).collect();
let requests =
cache_artifact_requests(self.session_id.clone(), artifacts);
let response = self.add_artifacts_rpc(requests).await?;
- Ok(Some((response, expected)))
+ Ok((status_ids, Some((response, expected))))
})
.await?;
+ self.verify_response(&status_ids.0, &status_ids.1)?;
if let Some((response, expected)) = uploaded {
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
check_artifact_upload(response, Some(&expected))?;
}
Ok(hashes)
@@ -1405,7 +1566,10 @@ impl SparkConnectClient {
// A read that failed ended the frames early, whatever the RPC
answered.
take_frame_failure(&failure)?;
match result {
- Ok(response) => return check_artifact_upload(response, None),
+ Ok(response) => {
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ return check_artifact_upload(response, None);
+ }
Err(status) => {
if self.retry_policy.can_retry(&status) {
if let Some(wait_ms) = state.next_attempt(None) {
@@ -1427,9 +1591,11 @@ impl SparkConnectClient {
) -> std::result::Result<spark_connect_proto::AddArtifactsResponse,
tonic::Status> {
let user_context = self.new_user_context();
let client_type = self.user_agent.clone();
+ let observed = std::sync::Arc::clone(&self.server_session_id);
let requests = requests.map(move |mut request| {
request.user_context = Some(user_context.clone());
request.client_type = Some(client_type.clone());
+ request.client_observed_server_side_session_id =
observed_of(&observed);
request
});
let deadline = self.rpc_deadlines.add_artifacts;
@@ -1450,33 +1616,37 @@ impl SparkConnectClient {
self.ensure_open()?;
let names: Vec<String> = names.iter().map(|n| n.to_string()).collect();
match self.artifact_status_rpc(&names).await {
- Ok(statuses) => Ok(statuses),
+ Ok(response) => {
+ self.verify_response(&response.session_id,
&response.server_side_session_id)?;
+ Ok(response
+ .statuses
+ .into_iter()
+ .map(|(name, status)| (name, status.exists))
+ .collect())
+ }
Err(status) => Err(self.error_from_status(status).await),
}
}
- /// The ArtifactStatus call of `names`, once: whether each artifact exists.
+ /// The ArtifactStatus call of `names`, once: the response, which says
whether each artifact
+ /// exists.
async fn artifact_status_rpc(
&self,
names: &[String],
- ) -> std::result::Result<std::collections::HashMap<String, bool>,
tonic::Status> {
+ ) -> std::result::Result<spark_connect_proto::ArtifactStatusesResponse,
tonic::Status> {
let mut request = ArtifactStatusesRequest::default();
request.session_id = self.session_id.clone();
request.user_context = Some(self.new_user_context());
request.client_type = Some(self.user_agent.clone());
request.names = names.to_vec();
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let deadline = self.rpc_deadlines.artifact_status;
let mut req = with_deadline(Request::new(request), deadline);
self._attach_metadata(&mut req);
- let response = within_deadline(deadline,
self.stub.clone().artifact_status(req))
- .await?
- .into_inner();
- Ok(response
- .statuses
- .into_iter()
- .map(|(name, status)| (name, status.exists))
- .collect())
+ within_deadline(deadline, self.stub.clone().artifact_status(req))
+ .await
+ .map(Response::into_inner)
}
/// Attach metadata headers to a request.
@@ -1578,7 +1748,8 @@ impl SparkConnectClient {
/// client's retry policy. Like the reference client, a failure is only
logged: the server
/// removes an execution that is never released once it has been detached
for
/// `spark.connect.execute.manager.detachedTimeout`.
- fn spawn_release(&self, metadata: MetadataMap, request:
ReleaseExecuteRequest) {
+ fn spawn_release(&self, metadata: MetadataMap, mut request:
ReleaseExecuteRequest) {
+ self.observe_on(&mut request.client_observed_server_side_session_id);
let mut stub = self.stub.clone();
let policy = self.retry_policy.clone();
let release = async move {
@@ -1660,6 +1831,12 @@ impl ReattachableResponseStream {
};
match next {
Ok(Some(resp)) => {
+ if let Err(error) = self
+ .client
+ .verify_response(&resp.session_id,
&resp.server_side_session_id)
+ {
+ return Err(self.fail(error));
+ }
self.iter.set_last_response_id(&resp).await;
if self.iter.is_completed().await {
self.done = true;
diff --git a/crates/spark-connect-core/tests/artifact_uploads.rs
b/crates/spark-connect-core/tests/artifact_uploads.rs
index 3f0cf9ac..62198a2c 100644
--- a/crates/spark-connect-core/tests/artifact_uploads.rs
+++ b/crates/spark-connect-core/tests/artifact_uploads.rs
@@ -437,6 +437,11 @@ fn acknowledge(frames: &[AddArtifactsRequest], corrupt:
bool) -> AddArtifactsRes
}
}
AddArtifactsResponse {
+ // The server echoes the session ID of the request.
+ session_id: frames
+ .first()
+ .map(|f| f.session_id.clone())
+ .unwrap_or_default(),
artifacts: artifacts
.into_iter()
.map(|(name, good)| ArtifactSummary {
diff --git a/crates/spark-connect-core/tests/deadlines.rs
b/crates/spark-connect-core/tests/deadlines.rs
index 9ec72340..f5631371 100644
--- a/crates/spark-connect-core/tests/deadlines.rs
+++ b/crates/spark-connect-core/tests/deadlines.rs
@@ -155,7 +155,7 @@ impl Fixture {
.unwrap();
});
let builder = ChannelBuilder::parse(&format!(
- "sc://{address}/;user_id=transport-user;token=fixture-token"
+
"sc://{address}/;user_id=transport-user;token=fixture-token;session_id=00000000-0000-4000-8000-000000000001"
))
.unwrap();
let mut client = bounded(SparkConnectClient::connect(&builder))
@@ -211,6 +211,8 @@ fn execute_request() -> ExecutePlanRequest {
fn response(id: &str, complete: bool) -> ExecutePlanResponse {
ExecutePlanResponse {
+ // The server echoes the session ID of the request.
+ session_id: execute_request().session_id,
response_id: id.into(),
response_type:
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
..Default::default()
diff --git a/crates/spark-connect-core/tests/reattach_cleanup.rs
b/crates/spark-connect-core/tests/reattach_cleanup.rs
index 9199853a..3fa59fdb 100644
--- a/crates/spark-connect-core/tests/reattach_cleanup.rs
+++ b/crates/spark-connect-core/tests/reattach_cleanup.rs
@@ -159,7 +159,7 @@ impl Fixture {
.unwrap();
});
let builder = ChannelBuilder::parse(&format!(
-
"sc://{address}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value"
+
"sc://{address}/;user_id=transport-user;token=fixture-token;x-cleanup=fixture-value;session_id=00000000-0000-4000-8000-000000000001"
))
.unwrap();
let mut client = bounded(SparkConnectClient::connect(&builder))
@@ -228,6 +228,8 @@ fn execute_request() -> ExecutePlanRequest {
fn response(id: &str, complete: bool) -> ExecutePlanResponse {
ExecutePlanResponse {
+ // The server echoes the session ID of the request.
+ session_id: "00000000-0000-4000-8000-000000000001".into(),
response_id: id.into(),
response_type:
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
..Default::default()
diff --git a/crates/spark-connect-core/tests/server_session.rs
b/crates/spark-connect-core/tests/server_session.rs
new file mode 100644
index 00000000..bcf92548
--- /dev/null
+++ b/crates/spark-connect-core/tests/server_session.rs
@@ -0,0 +1,505 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! The server-side session of a client, against a loopback gRPC service that
answers with the
+//! server-side session ID of the server, which a test changes to restart it:
the observed ID of
+//! the requests, the check of the responses, and the close of the client on a
restart.
+
+use std::collections::HashMap;
+use std::collections::VecDeque;
+use std::convert::Infallible;
+use std::pin::Pin;
+use std::sync::{Arc, Mutex};
+use std::task::{Context, Poll};
+
+use futures::{stream, Stream};
+use prost::Message;
+use spark_connect_core::bytes_codec::BytesCodec;
+use spark_connect_core::{ChannelBuilder, RetryPolicy, SparkConnectClient,
SparkErrorKind};
+use spark_connect_proto::execute_plan_response::{ResponseType, ResultComplete};
+use spark_connect_proto::{
+ ConfigRequest, ConfigResponse, ExecutePlanRequest, ExecutePlanResponse,
ReattachExecuteRequest,
+ ReleaseExecuteRequest, ReleaseExecuteResponse,
+};
+use tokio::net::TcpListener;
+use tonic::body::Body;
+use tonic::codegen::{http, BoxFuture, Service};
+use tonic::server::{NamedService, ServerStreamingService, UnaryService};
+use tonic::transport::Server;
+use tonic::{Code, Request, Response, Status};
+
+const SERVICE: &str = "/spark.connect.SparkConnectService/";
+const FIRST_SERVER_ID: &str = "server-1";
+const RESTARTED_SERVER_ID: &str = "server-2";
+type Frames = Pin<Box<dyn Stream<Item = Result<Vec<u8>, Status>> + Send>>;
+
+struct State {
+ /// The server-side session ID that the responses carry.
+ server_session_id: Mutex<String>,
+ /// Whether the next ExecutePlan streams its first response, and then
drops.
+ drop_next_execute: Mutex<bool>,
+ /// The errors that the next calls of a method fail with, in order.
+ failures: Mutex<VecDeque<(String, Status)>>,
+ /// The method and the bytes of every request, in the order they arrived.
+ requests: Mutex<Vec<(String, Vec<u8>)>>,
+}
+
+impl State {
+ fn record(&self, method: &str, request: Vec<u8>) {
+ self.requests
+ .lock()
+ .unwrap()
+ .push((method.to_string(), request));
+ }
+
+ fn requests_of<T: Message + Default>(&self, method: &str) -> Vec<T> {
+ self.requests
+ .lock()
+ .unwrap()
+ .iter()
+ .filter(|(m, _)| m == method)
+ .map(|(_, bytes)| T::decode(bytes.as_slice()).unwrap())
+ .collect()
+ }
+
+ fn server_session_id(&self) -> String {
+ self.server_session_id.lock().unwrap().clone()
+ }
+
+ fn failure(&self, method: &str) -> Option<Status> {
+ let mut failures = self.failures.lock().unwrap();
+ let index = failures.iter().position(|(m, _)| m == method)?;
+ failures.remove(index).map(|(_, status)| status)
+ }
+
+ fn take_drop_next_execute(&self) -> bool {
+ std::mem::take(&mut *self.drop_next_execute.lock().unwrap())
+ }
+}
+
+struct Fixture {
+ state: Arc<State>,
+ server: tokio::task::JoinHandle<()>,
+ address: std::net::SocketAddr,
+}
+
+impl Fixture {
+ async fn start() -> Self {
+ let state = Arc::new(State {
+ server_session_id: Mutex::new(FIRST_SERVER_ID.to_string()),
+ drop_next_execute: Mutex::new(false),
+ failures: Mutex::new(VecDeque::new()),
+ requests: Mutex::new(Vec::new()),
+ });
+ let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
+ let address = listener.local_addr().unwrap();
+ let incoming = futures::stream::unfold(listener, |listener| async {
+ Some((listener.accept().await.map(|(socket, _)| socket), listener))
+ });
+ let service = TestService(state.clone());
+ let server = tokio::spawn(async move {
+ Server::builder()
+ .add_service(service)
+ .serve_with_incoming(incoming)
+ .await
+ .unwrap();
+ });
+ Self {
+ state,
+ server,
+ address,
+ }
+ }
+
+ /// A client of the fixture, with a session of its own.
+ async fn client(&self) -> SparkConnectClient {
+ let builder = ChannelBuilder::parse(&format!("sc://{}",
self.address)).unwrap();
+ let mut client = SparkConnectClient::connect(&builder).await.unwrap();
+ client.set_retry_policy(RetryPolicy {
+ max_retries: Some(2),
+ initial_backoff_ms: 1,
+ max_backoff_ms: Some(2),
+ ..RetryPolicy::default()
+ });
+ client
+ }
+}
+
+impl Drop for Fixture {
+ fn drop(&mut self) {
+ self.server.abort();
+ }
+}
+
+/// The session ID of a request of `method`, which the server echoes in its
response.
+fn session_of(method: &str, bytes: &[u8]) -> String {
+ match method {
+ "ExecutePlan" => ExecutePlanRequest::decode(bytes).unwrap().session_id,
+ "ReattachExecute" =>
ReattachExecuteRequest::decode(bytes).unwrap().session_id,
+ "ReleaseExecute" =>
ReleaseExecuteRequest::decode(bytes).unwrap().session_id,
+ "Config" => ConfigRequest::decode(bytes).unwrap().session_id,
+ _ => String::new(),
+ }
+}
+
+/// The response of the whole result of an execution, as the server sends it.
+fn result_response(session_id: &str, server_session_id: &str) -> Vec<u8> {
+ ExecutePlanResponse {
+ session_id: session_id.into(),
+ server_side_session_id: server_session_id.into(),
+ response_id: "complete".into(),
+ response_type: Some(ResponseType::ResultComplete(ResultComplete {})),
+ ..Default::default()
+ }
+ .encode_to_vec()
+}
+
+/// The first response of an execution that drops before its result is
complete.
+fn first_response(session_id: &str, server_session_id: &str) -> Vec<u8> {
+ ExecutePlanResponse {
+ session_id: session_id.into(),
+ server_side_session_id: server_session_id.into(),
+ response_id: "first".into(),
+ ..Default::default()
+ }
+ .encode_to_vec()
+}
+
+#[derive(Clone)]
+struct TestService(Arc<State>);
+
+impl NamedService for TestService {
+ const NAME: &'static str = "spark.connect.SparkConnectService";
+}
+
+impl Service<http::Request<Body>> for TestService {
+ type Response = http::Response<Body>;
+ type Error = Infallible;
+ type Future = BoxFuture<Self::Response, Self::Error>;
+
+ fn poll_ready(&mut self, _: &mut Context<'_>) -> Poll<Result<(),
Self::Error>> {
+ Poll::Ready(Ok(()))
+ }
+
+ fn call(&mut self, request: http::Request<Body>) -> Self::Future {
+ let method =
request.uri().path().trim_start_matches(SERVICE).to_string();
+ let state = self.0.clone();
+ Box::pin(async move {
+ let mut grpc = tonic::server::Grpc::new(BytesCodec);
+ Ok(match method.as_str() {
+ "ExecutePlan" | "ReattachExecute" => {
+ grpc.server_streaming(Streamed(state, method), request)
+ .await
+ }
+ _ => grpc.unary(Unary(state, method), request).await,
+ })
+ })
+ }
+}
+
+/// ExecutePlan and ReattachExecute.
+struct Streamed(Arc<State>, String);
+
+impl ServerStreamingService<Vec<u8>> for Streamed {
+ type Response = Vec<u8>;
+ type ResponseStream = Frames;
+ type Future = BoxFuture<Response<Frames>, Status>;
+
+ fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
+ let bytes = request.into_inner();
+ self.0.record(&self.1, bytes.clone());
+ if let Some(status) = self.0.failure(&self.1) {
+ return Box::pin(async { Err(status) });
+ }
+ let session_id = session_of(&self.1, &bytes);
+ let server_session_id = self.0.server_session_id();
+ let items: Vec<Result<Vec<u8>, Status>> =
+ if self.1 == "ExecutePlan" && self.0.take_drop_next_execute() {
+ vec![
+ Ok(first_response(&session_id, &server_session_id)),
+ Err(Status::unavailable("stream disconnected")),
+ ]
+ } else {
+ vec![Ok(result_response(&session_id, &server_session_id))]
+ };
+ let frames: Frames = Box::pin(stream::iter(items));
+ Box::pin(async { Ok(Response::new(frames)) })
+ }
+}
+
+/// The unary RPCs that the tests use: Config and ReleaseExecute answer with
their IDs.
+struct Unary(Arc<State>, String);
+
+impl UnaryService<Vec<u8>> for Unary {
+ type Response = Vec<u8>;
+ type Future = BoxFuture<Response<Vec<u8>>, Status>;
+
+ fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
+ let bytes = request.into_inner();
+ self.0.record(&self.1, bytes.clone());
+ if let Some(status) = self.0.failure(&self.1) {
+ return Box::pin(async { Err(status) });
+ }
+ let session_id = session_of(&self.1, &bytes);
+ let server_side_session_id = self.0.server_session_id();
+ let body = match self.1.as_str() {
+ "Config" => ConfigResponse {
+ session_id,
+ server_side_session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ "ReleaseExecute" => ReleaseExecuteResponse {
+ session_id,
+ server_side_session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ _ => Vec::new(),
+ };
+ Box::pin(async { Ok(Response::new(body)) })
+ }
+}
+
+/// A `google.rpc.Status` with an `ErrorInfo` of `error_class`, as the server
sends the errors it
+/// raises.
+fn error_with_class(message: &str, error_class: &str) -> Status {
+ #[derive(Clone, PartialEq, prost::Message)]
+ struct ErrorInfo {
+ #[prost(string, tag = "1")]
+ reason: String,
+ #[prost(string, tag = "2")]
+ domain: String,
+ #[prost(map = "string, string", tag = "3")]
+ metadata: HashMap<String, String>,
+ }
+ #[derive(Clone, PartialEq, prost::Message)]
+ struct RpcStatus {
+ #[prost(int32, tag = "1")]
+ code: i32,
+ #[prost(string, tag = "2")]
+ message: String,
+ #[prost(message, repeated, tag = "3")]
+ details: Vec<prost_types::Any>,
+ }
+ let info = ErrorInfo {
+ reason: "SESSION_CHANGED".into(),
+ domain: "org.apache.spark".into(),
+ metadata: HashMap::from([("errorClass".to_string(),
error_class.to_string())]),
+ };
+ let details = RpcStatus {
+ code: Code::Internal as i32,
+ message: message.into(),
+ details: vec![prost_types::Any {
+ type_url: "type.googleapis.com/google.rpc.ErrorInfo".into(),
+ value: info.encode_to_vec(),
+ }],
+ }
+ .encode_to_vec();
+ Status::with_details(Code::Internal, message, details.into())
+}
+
+fn config_request(client: &SparkConnectClient) -> ConfigRequest {
+ ConfigRequest {
+ session_id: client.session_id().to_string(),
+ ..Default::default()
+ }
+}
+
+fn execute_request(client: &SparkConnectClient) -> ExecutePlanRequest {
+ ExecutePlanRequest {
+ session_id: client.session_id().to_string(),
+ ..Default::default()
+ }
+}
+
+/// Runs an execution of the client to its end, as the session does.
+async fn run_execution(client: &SparkConnectClient) {
+ let mut stream = client
+ .execute_plan_reattachable(execute_request(client))
+ .await
+ .unwrap();
+ while stream.message().await.unwrap().is_some() {}
+}
+
+#[tokio::test]
+async fn requests_carry_the_server_side_session_id_observed_in_the_responses()
{
+ let fixture = Fixture::start().await;
+ let client = fixture.client().await;
+
+ // No response has set the ID yet, so the first request carries none.
+ client.config(config_request(&client)).await.unwrap();
+ client.config(config_request(&client)).await.unwrap();
+ run_execution(&client).await;
+
+ // An execution that drops after its first response: the reattach takes
the observed ID of
+ // the execution, as the reference client's reattach does.
+ *fixture.state.drop_next_execute.lock().unwrap() = true;
+ run_execution(&client).await;
+
+ let configs = fixture.state.requests_of::<ConfigRequest>("Config");
+ assert_eq!(
+ configs
+ .iter()
+ .map(|r| r.client_observed_server_side_session_id.clone())
+ .collect::<Vec<_>>(),
+ [None, Some(FIRST_SERVER_ID.to_string())]
+ );
+ let executes = fixture
+ .state
+ .requests_of::<ExecutePlanRequest>("ExecutePlan");
+ assert_eq!(executes.len(), 2);
+ for request in &executes {
+ assert_eq!(
+ request.client_observed_server_side_session_id.as_deref(),
+ Some(FIRST_SERVER_ID)
+ );
+ }
+ let reattaches = fixture
+ .state
+ .requests_of::<ReattachExecuteRequest>("ReattachExecute");
+ assert_eq!(reattaches.len(), 1);
+ assert_eq!(
+ reattaches[0]
+ .client_observed_server_side_session_id
+ .as_deref(),
+ Some(FIRST_SERVER_ID)
+ );
+}
+
+#[tokio::test]
+async fn a_restarted_server_raises_and_closes_the_client() {
+ let fixture = Fixture::start().await;
+ let client = fixture.client().await;
+ let clone = client.clone();
+ client.config(config_request(&client)).await.unwrap();
+
+ // The server restarts, so its responses carry another server-side session
ID.
+ *fixture.state.server_session_id.lock().unwrap() =
RESTARTED_SERVER_ID.to_string();
+ let error = client
+ .config(config_request(&client))
+ .await
+ .expect_err("a response of a restarted server must raise");
+ assert_eq!(error.kind, SparkErrorKind::AssertionError);
+ assert_eq!(
+ error.message,
+ "Received incorrect server side session identifier for request. Please
create a new \
+ Spark Session to reconnect. (server-2 != server-1)"
+ );
+ assert!(client.is_closed() && clone.is_closed());
+
+ // The client is closed from then on, and so are its clones, as PySpark's
`_stub` raises.
+ let error = client
+ .config(config_request(&client))
+ .await
+ .expect_err("a closed client must raise");
+ assert_eq!(error.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+ let error = clone
+ .config(config_request(&clone))
+ .await
+ .expect_err("a clone of a closed client must raise");
+ assert_eq!(error.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+ let error = client
+ .execute_plan_reattachable(execute_request(&client))
+ .await
+ .err()
+ .expect("an execution of a closed client must raise");
+ assert_eq!(error.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+}
+
+#[tokio::test]
+async fn a_session_changed_error_closes_the_client() {
+ let fixture = Fixture::start().await;
+ let client = fixture.client().await;
+ fixture.state.failures.lock().unwrap().push_back((
+ "Config".to_string(),
+ error_with_class(
+ "The existing Spark server driver instance has restarted. Please
reconnect.",
+ "INVALID_HANDLE.SESSION_CHANGED",
+ ),
+ ));
+
+ let error = client
+ .config(config_request(&client))
+ .await
+ .expect_err("a session change must raise");
+ assert_eq!(
+ error.get_condition().as_deref(),
+ Some("INVALID_HANDLE.SESSION_CHANGED")
+ );
+ assert!(client.is_closed());
+
+ let error = client
+ .config(config_request(&client))
+ .await
+ .expect_err("a closed client must raise");
+ assert_eq!(error.get_condition().as_deref(), Some("NO_ACTIVE_SESSION"));
+}
+
+#[tokio::test]
+async fn
a_client_without_session_tracking_passes_the_ids_of_its_requests_through() {
+ // The transport-injection stub sends the requests of the reference
client's session, which
+ // tracks its server-side session itself, as over the grpcio stub that
does no tracking.
+ let fixture = Fixture::start().await;
+ let mut client = fixture.client().await;
+ client.set_session_tracking(false);
+ let request = ConfigRequest {
+ session_id: "the-reference-session".into(),
+ client_observed_server_side_session_id:
Some("observed-by-the-reference".into()),
+ ..Default::default()
+ };
+
+ // A response of another session than the client's is not an error.
+ client.config(request.clone()).await.unwrap();
+ // Nor is a response of a restarted server, which does not close the
client.
+ *fixture.state.server_session_id.lock().unwrap() =
RESTARTED_SERVER_ID.to_string();
+ client.config(request.clone()).await.unwrap();
+ // A session change is the error of its call, and does not close the
client either.
+ fixture.state.failures.lock().unwrap().push_back((
+ "Config".to_string(),
+ error_with_class(
+ "The existing Spark server driver instance has restarted. Please
reconnect.",
+ "INVALID_HANDLE.SESSION_CHANGED",
+ ),
+ ));
+ let error = client
+ .config(request.clone())
+ .await
+ .expect_err("the session change is the error of its call");
+ assert_eq!(
+ error.get_condition().as_deref(),
+ Some("INVALID_HANDLE.SESSION_CHANGED")
+ );
+ assert!(!client.is_closed());
+ client.config(request.clone()).await.unwrap();
+
+ // The requests keep the observed ID that the caller set. (The client
reads the stack trace
+ // confs of the error of the session change in a request of its own
session.)
+ let configs: Vec<ConfigRequest> = fixture
+ .state
+ .requests_of::<ConfigRequest>("Config")
+ .into_iter()
+ .filter(|config| config.session_id == "the-reference-session")
+ .collect();
+ assert_eq!(configs.len(), 4);
+ for config in &configs {
+ assert_eq!(
+ config.client_observed_server_side_session_id.as_deref(),
+ Some("observed-by-the-reference")
+ );
+ }
+}
diff --git a/crates/spark-connect-core/tests/user_context.rs
b/crates/spark-connect-core/tests/user_context.rs
index 09c9831d..590a518c 100644
--- a/crates/spark-connect-core/tests/user_context.rs
+++ b/crates/spark-connect-core/tests/user_context.rs
@@ -32,8 +32,9 @@ use spark_connect_core::{ChannelBuilder, RetryPolicy,
SparkConnectClient};
use spark_connect_proto::execute_plan_response::{ResponseType, ResultComplete};
use spark_connect_proto::release_execute_request::Release;
use spark_connect_proto::{
- AnalyzePlanRequest, ConfigRequest, ExecutePlanRequest, ExecutePlanResponse,
- FetchErrorDetailsRequest, InterruptRequest, ReattachExecuteRequest,
ReleaseExecuteRequest,
+ AnalyzePlanRequest, AnalyzePlanResponse, ConfigRequest, ConfigResponse,
ExecutePlanRequest,
+ ExecutePlanResponse, FetchErrorDetailsRequest, FetchErrorDetailsResponse,
InterruptRequest,
+ InterruptResponse, ReattachExecuteRequest, ReleaseExecuteRequest,
ReleaseExecuteResponse,
UserContext,
};
use tokio::net::TcpListener;
@@ -172,8 +173,10 @@ impl Drop for Fixture {
}
}
-fn response(id: &str, complete: bool) -> Vec<u8> {
+fn response(session_id: &str, id: &str, complete: bool) -> Vec<u8> {
ExecutePlanResponse {
+ // The server echoes the session ID of the request.
+ session_id: session_id.into(),
response_id: id.into(),
response_type:
complete.then_some(ResponseType::ResultComplete(ResultComplete {})),
..Default::default()
@@ -222,18 +225,22 @@ impl ServerStreamingService<Vec<u8>> for Streamed {
type Future = BoxFuture<Response<Frames>, Status>;
fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
- self.0.record(&self.1, request.into_inner());
+ let bytes = request.into_inner();
+ let session_id = session_of(&self.1, &bytes);
+ self.0.record(&self.1, bytes);
let execution = match self.1.as_str() {
"ExecutePlan" => self.0.executions.lock().unwrap().pop_front(),
_ => None,
};
let frames: Frames = match execution.unwrap_or(Execution::Complete) {
- Execution::Complete =>
Box::pin(stream::iter([Ok(response("complete", true))])),
- Execution::Open => {
- Box::pin(stream::iter([Ok(response("first",
false))]).chain(stream::pending()))
+ Execution::Complete => {
+ Box::pin(stream::iter([Ok(response(&session_id, "complete",
true))]))
}
+ Execution::Open => Box::pin(
+ stream::iter([Ok(response(&session_id, "first",
false))]).chain(stream::pending()),
+ ),
Execution::Disconnect => Box::pin(stream::iter([
- Ok(response("first", false)),
+ Ok(response(&session_id, "first", false)),
Err(Status::unavailable("stream disconnected")),
])),
};
@@ -249,8 +256,58 @@ impl UnaryService<Vec<u8>> for Unary {
type Future = BoxFuture<Response<Vec<u8>>, Status>;
fn call(&mut self, request: Request<Vec<u8>>) -> Self::Future {
- self.0.record(&self.1, request.into_inner());
- Box::pin(async { Ok(Response::new(Vec::new())) })
+ let bytes = request.into_inner();
+ let body = echo_response(&self.1, &bytes);
+ self.0.record(&self.1, bytes);
+ Box::pin(async { Ok(Response::new(body)) })
+ }
+}
+
+/// The session ID of a request of `method`, which the server echoes in its
response.
+fn session_of(method: &str, bytes: &[u8]) -> String {
+ match method {
+ "ExecutePlan" => ExecutePlanRequest::decode(bytes).unwrap().session_id,
+ "ReattachExecute" =>
ReattachExecuteRequest::decode(bytes).unwrap().session_id,
+ "AnalyzePlan" => AnalyzePlanRequest::decode(bytes).unwrap().session_id,
+ "Config" => ConfigRequest::decode(bytes).unwrap().session_id,
+ "Interrupt" => InterruptRequest::decode(bytes).unwrap().session_id,
+ "FetchErrorDetails" =>
FetchErrorDetailsRequest::decode(bytes).unwrap().session_id,
+ "ReleaseExecute" =>
ReleaseExecuteRequest::decode(bytes).unwrap().session_id,
+ _ => String::new(),
+ }
+}
+
+/// The response of a unary RPC, which echoes the session ID of its request;
the other fields
+/// are empty.
+fn echo_response(method: &str, bytes: &[u8]) -> Vec<u8> {
+ let session_id = session_of(method, bytes);
+ match method {
+ "AnalyzePlan" => AnalyzePlanResponse {
+ session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ "Config" => ConfigResponse {
+ session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ "Interrupt" => InterruptResponse {
+ session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ "FetchErrorDetails" => FetchErrorDetailsResponse {
+ session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ "ReleaseExecute" => ReleaseExecuteResponse {
+ session_id,
+ ..Default::default()
+ }
+ .encode_to_vec(),
+ _ => Vec::new(),
}
}
@@ -283,6 +340,7 @@ async fn send_typed_requests(
while stream.message().await.unwrap().is_some() {}
client
.analyze_plan(AnalyzePlanRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
..Default::default()
})
@@ -290,6 +348,7 @@ async fn send_typed_requests(
.unwrap();
client
.config(ConfigRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
..Default::default()
})
@@ -297,6 +356,7 @@ async fn send_typed_requests(
.unwrap();
client
.interrupt(InterruptRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
..Default::default()
})
@@ -304,6 +364,7 @@ async fn send_typed_requests(
.unwrap();
client
.fetch_error_details(FetchErrorDetailsRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
..Default::default()
})
@@ -311,6 +372,7 @@ async fn send_typed_requests(
.unwrap();
let mut stream = client
.reattach_execute(ReattachExecuteRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
operation_id: "typed-operation".into(),
..Default::default()
@@ -320,6 +382,7 @@ async fn send_typed_requests(
while stream.message().await.unwrap().is_some() {}
client
.release_execute(ReleaseExecuteRequest {
+ session_id: client.session_id().to_string(),
user_context: context.clone(),
operation_id: "typed-operation".into(),
..Default::default()
@@ -461,7 +524,10 @@ async fn
reattach_and_release_take_the_user_context_of_the_execution() {
let fixture = Fixture::start([Execution::Disconnect]).await;
let client = fixture.client(Some("configured")).await;
let mut stream = client
- .execute_plan_reattachable(ExecutePlanRequest::default())
+ .execute_plan_reattachable(ExecutePlanRequest {
+ session_id: client.session_id().to_string(),
+ ..Default::default()
+ })
.await
.unwrap();
let mut ids = Vec::new();
@@ -496,7 +562,10 @@ async fn
reattach_and_release_take_the_user_context_of_the_execution() {
let fixture = Fixture::start([Execution::Open]).await;
let client = fixture.client(Some("configured")).await;
let mut stream = client
- .execute_plan_reattachable(ExecutePlanRequest::default())
+ .execute_plan_reattachable(ExecutePlanRequest {
+ session_id: client.session_id().to_string(),
+ ..Default::default()
+ })
.await
.unwrap();
assert_eq!(
diff --git a/python/tests/_mock_connect_server.py
b/python/tests/_mock_connect_server.py
index 0d4beee0..4d38258e 100644
--- a/python/tests/_mock_connect_server.py
+++ b/python/tests/_mock_connect_server.py
@@ -286,6 +286,12 @@ def serve(connection):
self.ddl_string = ""
def analyze(self, request, context):
+ # AnalyzePlanRequest.session_id=1, echoed as
AnalyzePlanResponse.session_id=1, as the
+ # server does: the client checks it.
+ session_id = parse_message(request).get(1, [b""])[0]
+ return message_field(1, session_id) + self._analyze(request,
context)
+
+ def _analyze(self, request, context):
self.calls += 1
self.analyze_requests.append(request)
# base.proto: AnalyzePlanRequest.is_streaming=8;
AnalyzePlanResponse.is_streaming=6,
@@ -400,7 +406,10 @@ def serve(connection):
# ExecutePlanRequest.operation_id=6;
ExecutePlanResponse.operation_id=12,
# response_id=13, result_complete=14.
operation_id = parse_message(request).get(6, [b""])[0]
- header = message_field(12, operation_id)
+ # ExecutePlanRequest.session_id=1, echoed as
ExecutePlanResponse.session_id=1.
+ header = message_field(1, parse_message(request).get(1, [b""])[0])
+ message_field(
+ 12, operation_id
+ )
yield header + message_field(13, "r1") + self.execute_payload
yield header + message_field(13, "r2") + message_field(14, b"")
diff --git a/python/tests/test_dropin_offline.py
b/python/tests/test_dropin_offline.py
index fec6f738..fb11e3c5 100644
--- a/python/tests/test_dropin_offline.py
+++ b/python/tests/test_dropin_offline.py
@@ -14099,10 +14099,14 @@ def version_cache_serve(connection, request_started,
response_waiting, response_
)
if self.failure == "rpc":
context.abort(grpc.StatusCode.INVALID_ARGUMENT, "version
unavailable")
+ # AnalyzePlanRequest.session_id=1, echoed as
AnalyzePlanResponse.session_id=1.
+ session_id = parse_message(request).get(1, [b""])[0]
if self.failure == "no-result":
- return b""
+ return message_field(1, session_id)
# AnalyzePlanResponse.spark_version=8, SparkVersion.version=1.
- return version_cache_message(8, version_cache_message(1,
self.version))
+ return message_field(1, session_id) + version_cache_message(
+ 8, version_cache_message(1, self.version)
+ )
def release(self, request, context):
self.releases += 1
@@ -14112,7 +14116,10 @@ def version_cache_serve(connection, request_started,
response_waiting, response_
# CloneSessionResponse.new_session_id=3, of a new session.
import uuid
- return message_field(3, str(uuid.uuid4()))
+ # CloneSessionRequest.session_id=1, echoed as
CloneSessionResponse.session_id=1.
+ return message_field(1, parse_message(request).get(1, [b""])[0]) +
message_field(
+ 3, str(uuid.uuid4())
+ )
service = Service()
with version_cache_ThreadPoolExecutor(max_workers=4) as executor:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]