HyukjinKwon opened a new pull request, #324: URL: https://github.com/apache/spark-connect-rust/pull/324
### What changes were proposed in this pull request? [SPARK-61126](https://issues.apache.org/jira/browse/SPARK-61126): `DataFrame.pandas_api` and `DataFrame.plot` of the client are now those of PySpark 4.3.0. - **`pandas_api(index_col)`**: as PySpark's, it builds the `InternalFrame` of the DataFrame. The index is of the columns of `index_col` (`pyspark.pandas.namespace._get_index_map`). The result is the pandas-on-Spark DataFrame of that frame. - **`plot`**: as PySpark's, it is the `PySparkPlotAccessor` of `pyspark.sql.plot`, of the DataFrame itself. **This PR is stacked on #319 (SPARK-61120).** The accessor reads `DataFrame._session`, which #319 adds. Review the last commit, `[SPARK-61126]`; I will rebase this PR onto master once #319 merges. ### Why are the changes needed? - The client passed `index_col` to the constructor of `pyspark.pandas.DataFrame`, which does not take it. So `df.pandas_api(index_col="k")` raised `TypeError`. - The client's `plot` was the plot accessor of its pandas-on-Spark DataFrame, which differs from PySpark's accessor. Apache Spark's plot parity tests failed on this client. ### Does this PR introduce _any_ user-facing change? Yes: - `pandas_api(index_col=...)` gives a pandas-on-Spark DataFrame indexed by those columns. - `df.plot` is PySpark's `PySparkPlotAccessor`, as with PySpark 4.3.0. ### How was this patch tested? - A new offline test checks that `df.plot` is a `PySparkPlotAccessor` of the DataFrame itself. - A new live test checks: - the index and the columns of `pandas_api(index_col="k")`, and the index of `pandas_api()` - the figure of a line plot, if `plotly` is installed - Apache Spark's plot parity tests (`pyspark/sql/tests/connect/plot/`) run on this client against a local server, and all 18 pass. Before this change, 7 failed. - The lib tests of both crates (590 passed), the build of all tests, also with `wasm-udf`, the offline Python suites (2181 passed, 8 skipped), and the live drop-in and Arrow suites (177 passed) and live cases (10 passed) against a local Spark 4.3.0 server. Clippy reports no diagnostics in the changed lines. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
