Hyukjin Kwon created SPARK-61126:
------------------------------------
Summary: Make DataFrame.pandas_api take index_col, and
DataFrame.plot the PySparkPlotAccessor, in the Python client as PySpark does
Key: SPARK-61126
URL: https://issues.apache.org/jira/browse/SPARK-61126
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
Two methods of the DataFrame of the Python client differ from PySpark Connect's:
* {{DataFrame.pandas_api(index_col=None)}}: PySpark's builds the
{{InternalFrame}} of the DataFrame, of the index of the columns of
{{index_col}} ({{pyspark.pandas.namespace._get_index_map}}), as a
pandas-on-Spark DataFrame. The client passes {{index_col}} to the constructor
of {{pyspark.pandas.DataFrame}}, which takes none, so
{{sdf.pandas_api(index_col="Col1")}} raises {{TypeError: DataFrame.__init__()
got an unexpected keyword argument 'index_col'}}. Apache Spark's
{{test_pandas_api}} checks it.
* {{DataFrame.plot}}: PySpark's is the {{PySparkPlotAccessor}} of
{{pyspark.sql.plot}}, of the DataFrame itself. The client's is the plot
accessor of the pandas-on-Spark DataFrame of the DataFrame, of other options,
such as {{plotting.max_rows}} of pandas-on-Spark rather than
{{spark.sql.pyspark.plotting.max_rows}}.
The fix is to make them PySpark's.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
