[
https://issues.apache.org/jira/browse/SPARK-25829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663103#comment-16663103
]
Wenchen Fan commented on SPARK-25829:
-------------------------------------
After more thoughts, both the map lookup behavior and `Dataset.collect`
behavior are visible to end-users. It's hard to say which one is the official
semantic as there is no doc, and we have to do behavior change for one of them.
If we want to stick with the "earlier entry wins" semantic, then we need to fix
the 3 sub-tasks listed here.
If we want to stick with the "later entry wins" semantic, then we need to fix
the map lookup(GetMapValue) and other related functions like `map_filter`. And
for 2.4 we should revert these function if they are newly added, like
`map_filter`.
Any ideas? cc [~rxin] [~LI,Xiao] [~dongjoon] [~viirya] [~mgaido]
> Duplicated map keys are not handled consistently
> ------------------------------------------------
>
> Key: SPARK-25829
> URL: https://issues.apache.org/jira/browse/SPARK-25829
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.0
> Reporter: Wenchen Fan
> Priority: Major
>
> In Spark SQL, we apply "earlier entry wins" semantic to duplicated map keys.
> e.g.
> {code}
> scala> sql("SELECT map(1,2,1,3)[1]").show
> +------------------+
> |map(1, 2, 1, 3)[1]|
> +------------------+
> | 2|
> +------------------+
> {code}
> However, this handling is not applied consistently.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
