HyukjinKwon opened a new pull request, #313: URL: https://github.com/apache/spark-connect-rust/pull/313
### What changes were proposed in this pull request? [SPARK-61118](https://issues.apache.org/jira/browse/SPARK-61118): The CLI of Spark Declarative Pipelines works on this client, as with PySpark 4.3.0. - The CLI modules of `pyspark.pipelines` are vendored byte for byte from Apache Spark v4.3.0 (and added to the drift check): `cli` (`init`, `run` and `dry-run`), `init_cli`, `add_pipeline_analysis_context` and `block_session_mutations`. - `add_pipeline_analysis_context` adds a user context extension to the requests of a thread, through the client. As PySpark's `SparkConnectClient`, the client now keeps the user context extensions of each thread and the global ones: - `add_threadlocal_user_context_extension`, `add_global_user_context_extension`, `remove_user_context_extension` and `clear_user_context_extensions` - `thread_local.user_context_extensions` and `global_user_context_extensions`, of their IDs The Rust transport sends them with the requests of executions, analyses, configurations, error details and interrupts, as PySpark sends them: the global ones, then those of the thread. The core keeps them in its client, and its `SparkSession` gains the same methods. - The official tests of these modules are vendored as well (`test_cli`, `test_init_cli`, `test_add_pipeline_analysis_context`, `test_block_session_mutations`, `test_spark_connect` and `test_auto_cdc_flow`). A new step of the pipelines workflow runs them on this client's own classes against the server. The existing parity step runs official PySpark over this client's transport. ### Why are the changes needed? The `pyspark-client` package ships the CLI of Spark Declarative Pipelines. This client lacked its modules: `import pyspark.pipelines.cli` raised `ModuleNotFoundError`. The client also had no user context extensions, which the CLI adds for the analyses of a pipeline. ### Does this PR introduce _any_ user-facing change? Yes. `python -m pyspark.pipelines.cli init`, `run` and `dry-run` work as with PySpark 4.3.0. **For review:** as with `pyspark.ml` (SPARK-61111), the CLI needs `pyyaml` and `protobuf`. `pyspark-client` depends on both, while this PR does not add them to the dependencies of `pyspark-client-rust`. `docs/compatibility.md` says so. ### How was this patch tested? - Of the vendored official tests, 73 pass with this client against a local Spark 4.3.0 server. They cover the CLI, `init`, the analysis context, the blocking of session mutations, dry runs and auto CDC flows. - On a local server, `init`, a dry run and a run of the sample pipeline work. The blocking of session mutations raises the same errors as official PySpark 4.3.0. - New offline tests, against a mock Connect server, check: - the extensions that executions and analyses send, those of the thread and the global ones - that another thread sends the global ones only - the removal and the clearing of extensions - the extension of `add_pipeline_analysis_context`, only within its block - New Rust tests (`user_context_extensions.rs`): the extensions of each thread and the global ones, their removal, their clearing, and a new session of none. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2162 passed, 8 skipped), and the live drop-in and Arrow suites (172 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
