[
https://issues.apache.org/jira/browse/SPARK-58420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18100209#comment-18100209
]
Josh Rosen commented on SPARK-58420:
------------------------------------
Here's a somewhat-minimalistic pure SQL repro:
{code:sql}
-- Two maps that are semantically equal (they differ only in key insertion
order):
SELECT count(DISTINCT m)
FROM VALUES (map('a',1,'b',2)), (map('b',2,'a',1)) AS t(m);
-- outputs: 2 WRONG (expected: 1)
-- GROUP BY on the same data handles it correctly (MapSort is injected there):
SELECT count(*)
FROM (SELECT m FROM VALUES (map('a',1,'b',2)), (map('b',2,'a',1)) AS t(m) GROUP
BY m);
-- outputs: 1 correct
{code}
> COUNT(DISTINCT ...) does not sort maps in the argument trees
> ------------------------------------------------------------
>
> Key: SPARK-58420
> URL: https://issues.apache.org/jira/browse/SPARK-58420
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.3.0
> Reporter: Vladimir Golubev
> Priority: Major
> Labels: correctness
>
> Injecting MapSort is necessary for maps to be grouped correctly. We currently
> do that for GROUP BY, but we missed that for the COUNT(DISTINCT ...)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
