Hyukjin Kwon created SPARK-61225:
------------------------------------
Summary: Collecting a map whose key type is a user-defined type
fails with TypeError
Key: SPARK-61225
URL: https://issues.apache.org/jira/browse/SPARK-61225
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
collect(), toLocalIterator(), head(), first() and take() of a DataFrame with a
MapType whose key is a UDT (for example MapType(PointUDT(), IntegerType()))
raise "TypeError: unhashable type: 'list'" in the Python client. The Rust side
builds the dict from the raw storage key (a list) before the UDT converter
runs. PySpark deserializes each key with the UDT first and returns {(1.0, 2.0):
1}. Map values of a UDT already work. Maps nested in arrays, structs and map
values are affected too. Plain array-keyed maps raise the same TypeError in
both clients and are unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
