HyukjinKwon opened a new pull request, #238: URL: https://github.com/apache/spark-connect-rust/pull/238
### What changes were proposed in this pull request? [SPARK-61036](https://issues.apache.org/jira/browse/SPARK-61036): `cloneSession` now clones the session on the server, as PySpark Connect does. `spark.cloneSession(new_session_id=None)` sends a `CloneSession` RPC. The new session has a copy of the server-side state of this one, its SQL configurations, temporary views, registered functions and catalog state. It has the id `new_session_id` if one is given, which must be a UUID. Changes to either session after that do not show in the other. In the core, `SparkConnectClient::clone_session` sends the RPC under PySpark's `clone_session` deadline of 10 minutes. `SparkSession::clone_session` and the new `clone_session_with_id` return the clone. `SparkSession::clone_session` now returns a `Result`, as the RPC can fail. The mock Connect servers of the tests answer `CloneSession` requests. ### Why are the changes needed? The client returned a new session instead, with none of the state of this one, and ignored `new_session_id`. After `spark.conf.set("spark.sql.shuffle.partitions", "7")`, `spark.cloneSession().conf.get("spark.sql.shuffle.partitions")` was `200`, not `7`, and the session's temporary views were not found. ### Does this PR introduce _any_ user-facing change? Yes. `cloneSession` clones the session, as in PySpark 4.3.0. For an invalid `new_session_id`, the client raises the server's `INVALID_CLONE_SESSION_REQUEST.TARGET_SESSION_ID_FORMAT` error; PySpark 4.3.0 raises the gRPC error itself. In the Rust API, `SparkSession::clone_session` returns a `Result<SparkSession>`. ### How was this patch tested? - A new live test checks the configurations, temporary views and session ids of clones, and their isolation. It passes with official PySpark 4.3.0, and fails without this change. - A new e2e test checks the same of the Rust API against the server, and the error of an invalid id. - A new offline test checks the `CloneSession` requests sent to a mock Connect server. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1810 passed, 8 skipped), and the live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
