HyukjinKwon opened a new pull request, #283: URL: https://github.com/apache/spark-connect-rust/pull/283
### What changes were proposed in this pull request? [SPARK-61086](https://issues.apache.org/jira/browse/SPARK-61086): `SparkSession.Builder` now sets its options on itself and validates its startup URLs, as PySpark's does. As PySpark's: - `SparkSession.builder` returns a new builder on each access (PySpark's `classproperty`). - `config`, `master`, `remote`, `appName` and the other methods set their option on the builder and return the same builder. So an option set without chaining is set on the session of `getOrCreate`: ```python builder = SparkSession.builder.remote(url) builder.config("spark.sql.shuffle.partitions", "7") spark = builder.getOrCreate() ``` - After each option, the builder validates the startup URLs: - a master (`spark.master`, or else `MASTER`) with a remote (`spark.remote`, or else `SPARK_REMOTE`) raises the `PySparkRuntimeError` `CANNOT_CONFIGURE_SPARK_CONNECT_MASTER` - with `SPARK_LOCAL_REMOTE` set, a remote that differs from `SPARK_REMOTE` and is not local raises `CANNOT_CONFIGURE_SPARK_CONNECT` - The class is `SparkSession.Builder`: its name is `Builder` and its `__qualname__` is `SparkSession.Builder`. ### Why are the changes needed? - Each method of the builder returned a new builder, so an option set without chaining was lost. - The class was named `SparkSessionBuilder`. - The builder accepted a master together with a remote, and `master()` ignored its value. ### Does this PR introduce _any_ user-facing change? Yes. The builder sets and validates its options as in PySpark 4.3.0. Code that sets a master together with a remote now raises, as with PySpark. ### How was this patch tested? - A new live test sets an option on a builder without chaining, and checks it on the session of `getOrCreate`. It passes with official PySpark 4.3.0 and fails without this change. - A new offline test checks the new builder of each access, the returned builder, the qualified name, and the errors of a master with a remote, also from `MASTER`, `SPARK_REMOTE` and `SPARK_LOCAL_REMOTE`. - Older offline tests checked the name `SparkSessionBuilder`, or set a master together with a remote; they now check `Builder`, and no longer set a master with a remote. - 20 cases of the builder, of its returns, its class and its URL errors, give the same results as official PySpark 4.3.0. - The lib tests of both crates (592 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2026 passed, 12 skipped), and the live drop-in and Arrow suites (142 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
