[
https://issues.apache.org/jira/browse/SPARK-60126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibo Dong updated SPARK-60126:
------------------------------
Description:
h2. What happened
Spark SQL's {{mode}} aggregate can return a less frequent {{MAP}} value even
when another value is the unique most frequent value. The same query can
produce different results, including incorrect ones.
h2. How to reproduce
Run the following SQL using {{{}spark-sql{}}}:
{code:sql}
SELECT g, mode(m) AS m
FROM (
SELECT g,
CASE WHEN id <= 3
THEN map(g, 'a', g + 100, 'b')
ELSE map(g, 'x')
END AS m
FROM VALUES
(0, 1), (0, 2), (0, 3), (0, 4),
(1, 1), (1, 2), (1, 3), (1, 4),
(2, 1), (2, 2), (2, 3), (2, 4),
(3, 1), (3, 2), (3, 3), (3, 4),
(4, 1), (4, 2), (4, 3), (4, 4),
(5, 1), (5, 2), (5, 3), (5, 4),
(6, 1), (6, 2), (6, 3), (6, 4),
(7, 1), (7, 2), (7, 3), (7, 4)
AS t(g, id)
) q
GROUP BY g
ORDER BY g;
{code}
Save the SQL as {{repro.sql}} and run:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.shuffle.partitions=2
-f repro.sql
{code}
h2. Expected result
For each group, the two-entry map occurs three times, while the one-entry map
occurs only once. Therefore, {{mode(m)}} should always return the two-entry map.
{code}
g | m
--+----------------------
0 | {0 -> a, 100 -> b}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
h2. Actual result
The query can return a one-entry map that occurs only once, instead of the most
frequent map. For example:
{code}
Unable to find source-code formatter for language: text. Available languages
are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl,
php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yamlg |
m
--+----------------------
0 | {0 -> x}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
In this result, {{mode(m)}} selects {{0 -> x}} for group {{{}0{}}}, although
that map occurs only once and the other map occurs three times.
was:
h2. What happened
Spark SQL's {{mode}} aggregate can return a less frequent {{MAP}} value even
when another value is the unique most frequent value. The same query can
produce different results, including incorrect ones.
h2. How to reproduce
Run the following SQL using {{spark-sql}}:
{code:sql}
SELECT g, mode(m) AS m
FROM (
SELECT g,
CASE WHEN id <= 3
THEN map(g, 'a', g + 100, 'b')
ELSE map(g, 'x')
END AS m
FROM VALUES
(0, 1), (0, 2), (0, 3), (0, 4),
(1, 1), (1, 2), (1, 3), (1, 4),
(2, 1), (2, 2), (2, 3), (2, 4),
(3, 1), (3, 2), (3, 3), (3, 4),
(4, 1), (4, 2), (4, 3), (4, 4),
(5, 1), (5, 2), (5, 3), (5, 4),
(6, 1), (6, 2), (6, 3), (6, 4),
(7, 1), (7, 2), (7, 3), (7, 4)
AS t(g, id)
) q
GROUP BY g
ORDER BY g;
{code}
Save the SQL as {{repro.sql}} and run:
{code:bash}
spark-sql --master 'local[2]'
--conf spark.ui.enabled=false
--conf spark.sql.shuffle.partitions=2
-f repro.sql
{code}
h2. Expected result
For each group, the two-entry map occurs three times, while the one-entry map
occurs only once. Therefore, {{mode(m)}} should always return the two-entry map.
{code:text}
g | m
--+----------------------
0 | {0 -> a, 100 -> b}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
h2. Actual result
The query can return a one-entry map that occurs only once, instead of the most
frequent map. For example:
{code:text}
g | m
--+----------------------
0 | {0 -> x}
1 | {1 -> a, 101 -> b}
2 | {2 -> a, 102 -> b}
3 | {3 -> a, 103 -> b}
4 | {4 -> a, 104 -> b}
5 | {5 -> a, 105 -> b}
6 | {6 -> a, 106 -> b}
7 | {7 -> a, 107 -> b}
{code}
In this result, {{mode(m)}} selects {{0 -> x}} for group {{0}}, although that
map occurs only once and the other map occurs three times.
> mode can return a less frequent value for MAP inputs
> ----------------------------------------------------
>
> Key: SPARK-60126
> URL: https://issues.apache.org/jira/browse/SPARK-60126
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Environment: Spark: 5.0.0-SNAPSHOT
> Commit: 38422aefaf599ffb4cfa579d98898d95b1f02081
> Java: OpenJDK 17.0.17
> OS: macOS 15.7.4
> Reporter: Yibo Dong
> Priority: Major
>
> h2. What happened
> Spark SQL's {{mode}} aggregate can return a less frequent {{MAP}} value even
> when another value is the unique most frequent value. The same query can
> produce different results, including incorrect ones.
> h2. How to reproduce
> Run the following SQL using {{{}spark-sql{}}}:
> {code:sql}
> SELECT g, mode(m) AS m
> FROM (
> SELECT g,
> CASE WHEN id <= 3
> THEN map(g, 'a', g + 100, 'b')
> ELSE map(g, 'x')
> END AS m
> FROM VALUES
> (0, 1), (0, 2), (0, 3), (0, 4),
> (1, 1), (1, 2), (1, 3), (1, 4),
> (2, 1), (2, 2), (2, 3), (2, 4),
> (3, 1), (3, 2), (3, 3), (3, 4),
> (4, 1), (4, 2), (4, 3), (4, 4),
> (5, 1), (5, 2), (5, 3), (5, 4),
> (6, 1), (6, 2), (6, 3), (6, 4),
> (7, 1), (7, 2), (7, 3), (7, 4)
> AS t(g, id)
> ) q
> GROUP BY g
> ORDER BY g;
> {code}
> Save the SQL as {{repro.sql}} and run:
> {code:bash}
> spark-sql --master 'local[2]'
> --conf spark.ui.enabled=false
> --conf spark.sql.shuffle.partitions=2
> -f repro.sql
> {code}
> h2. Expected result
> For each group, the two-entry map occurs three times, while the one-entry map
> occurs only once. Therefore, {{mode(m)}} should always return the two-entry
> map.
> {code}
> g | m
> --+----------------------
> 0 | {0 -> a, 100 -> b}
> 1 | {1 -> a, 101 -> b}
> 2 | {2 -> a, 102 -> b}
> 3 | {3 -> a, 103 -> b}
> 4 | {4 -> a, 104 -> b}
> 5 | {5 -> a, 105 -> b}
> 6 | {6 -> a, 106 -> b}
> 7 | {7 -> a, 107 -> b}
> {code}
> h2. Actual result
> The query can return a one-entry map that occurs only once, instead of the
> most frequent map. For example:
> {code}
> Unable to find source-code formatter for language: text. Available languages
> are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go,
> groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc,
> perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml,
> yamlg | m
> --+----------------------
> 0 | {0 -> x}
> 1 | {1 -> a, 101 -> b}
> 2 | {2 -> a, 102 -> b}
> 3 | {3 -> a, 103 -> b}
> 4 | {4 -> a, 104 -> b}
> 5 | {5 -> a, 105 -> b}
> 6 | {6 -> a, 106 -> b}
> 7 | {7 -> a, 107 -> b}
> {code}
> In this result, {{mode(m)}} selects {{0 -> x}} for group {{{}0{}}}, although
> that map occurs only once and the other map occurs three times.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
