ueshin opened a new pull request, #40170:
URL: https://github.com/apache/spark/pull/40170
### What changes were proposed in this pull request?
Fixes `DataFrame.toPandas` to handle duplicated column names.
### Why are the changes needed?
Currently
```py
spark.sql("select 1 v, 1 v").toPandas()
```
fails with the error:
```py
Traceback (most recent call last):
...
File ".../python/pyspark/sql/connect/dataframe.py", line 1335, in toPandas
return self._session.client.to_pandas(query)
File ".../python/pyspark/sql/connect/client.py", line 548, in to_pandas
pdf = table.to_pandas()
File "pyarrow/array.pxi", line 830, in
pyarrow.lib._PandasConvertible.to_pandas
File "pyarrow/table.pxi", line 3908, in pyarrow.lib.Table._to_pandas
File "/.../lib/python3.9/site-packages/pyarrow/pandas_compat.py", line
819, in table_to_blockmanager
columns = _deserialize_column_index(table, all_columns, column_indexes)
File "/.../lib/python3.9/site-packages/pyarrow/pandas_compat.py", line
938, in _deserialize_column_index
columns = _flatten_single_level_multiindex(columns)
File "/.../lib/python3.9/site-packages/pyarrow/pandas_compat.py", line
1186, in _flatten_single_level_multiindex
raise ValueError('Found non-unique column index')
ValueError: Found non-unique column index
```
Simliar to #28210.
### Does this PR introduce _any_ user-facing change?
Duplicated column names will be available when calling `toPandas()`.
### How was this patch tested?
Enabled related tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
