HyukjinKwon opened a new pull request, #247: URL: https://github.com/apache/spark-connect-rust/pull/247
### What changes were proposed in this pull request? [SPARK-61047](https://issues.apache.org/jira/browse/SPARK-61047): `spark.conf` now warns the warnings of the server, and takes its arguments, as PySpark Connect's `RuntimeConf` does. - `spark.conf.set`, `spark.conf.unset` and the options of `SparkSession.builder.config` warn each warning of the server as a `UserWarning`, such as "The SQL config 'spark.sql.execution.arrow.enabled' is NOT supported in Spark Connect". - `spark.conf.get` raises `NOT_EXPECTED_TYPE` of `key` for a key that is not a str. It raises it of `default` for a default that is neither None nor a str. - `spark.conf.set` sends: - a bool as "true" or "false" - an int as its `str()` - a str, or UTF-8 bytes, as it is - None as no value, which the server rejects It raises the `TypeError` of protobuf for a value of another type, such as a float. The core `RuntimeConf::set`, `set_all` and `unset`, and the `SparkConnectClient` methods they call, now return the warnings of the server. The new `RuntimeConf::set_pairs` takes values that may be None. The builder warns its warnings once. PySpark warns them twice when it creates a session, as it sets its options twice; under the default warning filter, both show one. ### Why are the changes needed? The client: - dropped the warnings of the server, so a deprecated or unsupported configuration was set silently - raised `TypeError`s for the arguments of `get` - set the `str()` of any value: `spark.conf.set(key, None)` set "null", and `spark.conf.set(key, 1.5)` set "1.5" ### Does this PR introduce _any_ user-facing change? Yes. `spark.conf` warns, and takes values, as in PySpark 4.3.0. ### How was this patch tested? - A new live test checks the warnings of `set` and `unset` of an unsupported configuration, the rejection of None by the server, the `TypeError` of a float, and the errors of `get`. It passes with official PySpark 4.3.0, and fails without this change. - A new offline test checks the warnings of `set`, `unset` and the builder, the values that `set` sends, and the errors of `set` and `get`, against a mock Connect server. The mock server now answers `Unset`, returns the warnings of a test, and records a value that is not set. - On a local server, 42 cases of `spark.conf` give the results, errors and warnings of official PySpark 4.3.0. The exceptions are the warnings of the builder, as above, and the messages of the `TypeError`s of 3 keys that are not a str. - The lib tests of both crates (571 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (1811 passed, 8 skipped), and the live drop-in and Arrow suites (109 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
