[
https://issues.apache.org/jira/browse/SPARK-61126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-61126.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 324
[https://github.com/apache/spark-connect-rust/pull/324]
> Make DataFrame.pandas_api take index_col, and DataFrame.plot the
> PySparkPlotAccessor, in the Python client as PySpark does
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61126
> URL: https://issues.apache.org/jira/browse/SPARK-61126
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> Two methods of the DataFrame of the Python client differ from PySpark
> Connect's:
> * {{DataFrame.pandas_api(index_col=None)}}: PySpark's builds the
> {{InternalFrame}} of the DataFrame, of the index of the columns of
> {{index_col}} ({{pyspark.pandas.namespace._get_index_map}}), as a
> pandas-on-Spark DataFrame. The client passes {{index_col}} to the constructor
> of {{pyspark.pandas.DataFrame}}, which takes none, so
> {{sdf.pandas_api(index_col="Col1")}} raises {{TypeError: DataFrame.__init__()
> got an unexpected keyword argument 'index_col'}}. Apache Spark's
> {{test_pandas_api}} checks it.
> * {{DataFrame.plot}}: PySpark's is the {{PySparkPlotAccessor}} of
> {{pyspark.sql.plot}}, of the DataFrame itself. The client's is the plot
> accessor of the pandas-on-Spark DataFrame of the DataFrame, of other options,
> such as {{plotting.max_rows}} of pandas-on-Spark rather than
> {{spark.sql.pyspark.plotting.max_rows}}.
> The fix is to make them PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
