Hyukjin Kwon created SPARK-61146:
------------------------------------
Summary: Create DataFrames of pandas frames whose struct fields
repeat names in the Python client, as PySpark does
Key: SPARK-61146
URL: https://issues.apache.org/jira/browse/SPARK-61146
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect's createDataFrame of a pandas DataFrame under a schema whose
structs repeat field names, such as struct<x: string, x: int>, converts the
frame with the deduplicated names (x_0, x_1) of _deduplicate_field_names, and
sends that Arrow table with the schema as given, which the server applies.
The Python client converts the frame the same way, but then normalizes the
table again as an Arrow table of the caller, which may not repeat the field
names of a struct, and raises:
UnsupportedOperationException: [DUPLICATED_FIELD_NAME_IN_ARROW_STRUCT]
Duplicated field names in Arrow Struct are not allowed, got ['x', 'x']
Apache Spark's test_createDataFrame_pandas_duplicate_field_names of
test_parity_arrow.py fails on this client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
