Hyukjin Kwon created SPARK-60927:
------------------------------------
Summary: Keep the keys of map values typed and in order in Spark
Connect Rust
Key: SPARK-60927
URL: https://issues.apache.org/jira/browse/SPARK-60927
Project: Spark
Issue Type: Bug
Components: Connect
Affects Versions: connect-rust-4.3.0
Reporter: Hyukjin Kwon
PySpark Connect reads a map value into a dict of its entries, in the order that
the server sent them, with keys of their own type. The Rust-backed Python
client keeps a map value as string keys, sorted, so these PySpark 4.3.0 results
differ:
* {{spark.sql("SELECT map(1, 'a', 2, 'b') AS m").first().m}} has the integer
keys 1 and 2. The client gives the string keys '1' and '2'; a boolean, double
or date key also becomes a string, a date as its day number.
* {{map('b', 1, 'a', 2)}} keeps the key 'b' first. The client sorts the keys,
so 'a' comes first.
* {{map(array(1), 'x')}} raises {{TypeError}}, as a list key is unhashable. The
client gives the key in a Rust debug form, {{'List([Integer(1)])'}}.
* {{createDataFrame}} of a dict with integer keys, such as {{[(\{1: 'a'\},)]}},
infers {{map<bigint,string>}}. The client sends {{map<string,string>}}.
Keep the entries of a map value in order, with keys of their own type, in the
Rust core and in the Python client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
