HyukjinKwon opened a new pull request, #319: URL: https://github.com/apache/spark-connect-rust/pull/319
### What changes were proposed in this pull request? [SPARK-61120](https://issues.apache.org/jira/browse/SPARK-61120): Modules vendored from Apache Spark v4.3.0 read private attributes of PySpark Connect's DataFrame, session and client. This PR adds them: - `DataFrame._session`, the session of a DataFrame. These read it: - `pyspark.ml.util._cache_spark_dataset`, which `CrossValidator` and `TrainValidationSplit` use with `SPARKML_TEMP_DFS_PATH` - `pyspark.sql.plot`, for the maximum rows of a plot - `select_dtypes` of `pyspark.pandas` - `SparkSession._client`, the client of `SparkSession.client`, and `SparkConnectClient._resources()`, the resources of the cluster of a `GetResourcesCommand`. `TorchDistributor` reads them on Spark Connect. - `SparkConnectClient.execute_command` now converts the results as PySpark's does: - a `GetResourcesCommand` gives a dict of `ResourceInformation` - a resource profile gives its profile ID - a checkpoint gives a DataFrame of its cached relation It also returns the `pipeline_event_result` of a response. - `SparkConnectClient.release_session()`, which releases the session on the server, and `close()`, which closes the client. A closed client `is_closed`, and its session is stopped. Apache Spark's tests call them to clean up a new session. `dev/parity/known-differences.md` records why two of Apache Spark's ML parity tests still fail. `MockDataset` of `pyspark.testing.mlutils` creates itself with `object.__new__`, which a subclass of a class of the Rust extension refuses. ### Why are the changes needed? - On the client, `df._session` was a Column of that name. So with `SPARKML_TEMP_DFS_PATH` set, `CrossValidator(...).fit(df)` raised `TypeError: 'Column' object is not callable`. - `TorchDistributor` failed on `session._client`. ### Does this PR introduce _any_ user-facing change? Yes: - `CrossValidator` and `TrainValidationSplit` work with `SPARKML_TEMP_DFS_PATH`. - `TorchDistributor` reads the resources of the cluster. - `execute_command` returns its results as PySpark's does. ### How was this patch tested? - New offline tests, against a mock Connect server, check: - `DataFrame._session` and `SparkSession._client` - the conversions of the results of the resources, a resource profile, a checkpoint (whose relation is removed from the server once the DataFrame is collected) and a pipeline event - `_resources()` - New live tests: - one fits a `CrossValidator` with `SPARKML_TEMP_DFS_PATH` set, and reads the resources of the server - one releases and closes the client of a new session They pass with official PySpark 4.3.0 and fail without this change. - Apache Spark's `test_parity_sql_context.py` passes on this client against a local server, all 21 tests. Before this change, `test_newSession_returns_distinct_instance` failed. - Apache Spark's `pyspark/ml/tests/connect/test_parity_*.py` run on this client against a local server. Apart from the two tests of `MockDataset` and one that reads Spark's sample data from the server's working directory, they pass. That includes `test_cross_validator` and `test_train_validation_split`, which failed without this change. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2180 passed, 8 skipped), and the live drop-in and Arrow suites (176 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
