HyukjinKwon opened a new pull request, #211: URL: https://github.com/apache/spark-connect-rust/pull/211
### What changes were proposed in this pull request? [SPARK-61008](https://issues.apache.org/jira/browse/SPARK-61008): `SparkSession.builder` now gets or creates sessions as PySpark 4.3.0's does. - `getOrCreate()` returns the active session of the thread, or else the default session, as the same object, and creates one only when there is neither. It sets the options of the builder on the session that it returns. - The options are set in one request whose failures are ignored, as `_apply_options` sends. That ignores, for example, static configurations. `spark.remote` and `spark.master` are not options, and `appName()` sets `spark.app.name`. - The URL is that of `remote()`, or else of `SPARK_REMOTE`. Without either, `PySparkRuntimeError` `CONNECT_URL_NOT_SET` is raised. - The first session created is the default session, and the active session is thread-local. `create()` makes a new session, which is the default and the active session only when they are not set. `stop()` unsets them, and `SparkSession.active()` returns the active or else the default session. The core configuration has `set_all`, of several values in one request, silently or not, for this. ### Why are the changes needed? `getOrCreate()` created a new session each time, so the configurations and temporary views of a session were not those of the next one. After `spark.conf.set("spark.sql.shuffle.partitions", "7")` and `createOrReplaceTempView("v")`, a second `getOrCreate()` returned a session whose configuration was `200` and that had no view `v`. A library that calls `getOrCreate()` got a session of its own. `getOrCreate()` also raised `ValueError` without `remote()`, even with `SPARK_REMOTE` set. One active session was kept for all threads, and `create()` replaced it. ### Does this PR introduce _any_ user-facing change? Yes. `getOrCreate()`, `create()`, `getActiveSession()` and `active()` behave as in PySpark 4.3.0, and an existing session is returned with the options of the builder set on it. ### How was this patch tested? - A new offline test checks, in a new interpreter: - `getOrCreate()` returns the active session, as the same object, whatever the URL, and the default session in another thread. - `create()` does not replace the active session. - `stop()` unsets the sessions. - The URL comes from `SPARK_REMOTE`, and `CONNECT_URL_NOT_SET` and `NO_ACTIVE_OR_DEFAULT_SESSION` are raised. - A new offline test checks the options that a mock Connect server receives. They arrive in one silent request, without `spark.master`, and once for an existing session too. The mock now serves Config `Set` requests. - A new live test checks that `getOrCreate()` returns the session with an option set, and that a static configuration is ignored. It also passes with official PySpark 4.3.0. - The new tests fail without this change. - The lib tests and every integration test, e2e included, against a local Spark 4.3.0 server (664 passed, and 125 of the transport crate), the build of all tests with `wasm-udf`, the offline Python suites (1161 passed, 8 skipped), and the live drop-in and Arrow suites (87 passed) and live cases (10 passed). Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
