[
https://issues.apache.org/jira/browse/SPARK-60927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-60927.
----------------------------------
Resolution: Fixed
Issue resolved by pull request 161
[https://github.com/apache/spark-connect-rust/pull/161]
> Keep the keys of map values typed and in order in Spark Connect Rust
> --------------------------------------------------------------------
>
> Key: SPARK-60927
> URL: https://issues.apache.org/jira/browse/SPARK-60927
> Project: Spark
> Issue Type: Bug
> Components: Connect
> Affects Versions: connect-rust-4.3.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: connect-rust-4.3.0
>
>
> PySpark Connect reads a map value into a dict of its entries, in the order
> that the server sent them, with keys of their own type. The Rust-backed
> Python client keeps a map value as string keys, sorted, so these PySpark
> 4.3.0 results differ:
> * {{spark.sql("SELECT map(1, 'a', 2, 'b') AS m").first().m}} has the integer
> keys 1 and 2. The client gives the string keys '1' and '2'; a boolean, double
> or date key also becomes a string, a date as its day number.
> * {{map('b', 1, 'a', 2)}} keeps the key 'b' first. The client sorts the keys,
> so 'a' comes first.
> * {{map(array(1), 'x')}} raises {{TypeError}}, as a list key is unhashable.
> The client gives the key in a Rust debug form, {{'List([Integer(1)])'}}.
> * {{createDataFrame}} of a dict with integer keys, such as {{[(\{1:
> 'a'\},)]}}, infers {{map<bigint,string>}}. The client sends
> {{map<string,string>}}.
> Keep the entries of a map value in order, with keys of their own type, in the
> Rust core and in the Python client.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
