[
https://issues.apache.org/jira/browse/SPARK-58959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107192#comment-18107192
]
Eric Yang commented on SPARK-58959:
-----------------------------------
I'm working on the fix.
> Duplicate PIVOT values corrupt the aggregation buffer on the PivotFirst fast
> path
> ---------------------------------------------------------------------------------
>
> Key: SPARK-58959
> URL: https://issues.apache.org/jira/browse/SPARK-58959
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Eric Yang
> Priority: Major
>
> When a PIVOT ... IN list contains two values that compare equal, the optimized
> PivotFirst fast path writes outside the slots it owns in the aggregation
> buffer.
> Repro:
>
>
> {code:java}
> CREATE OR REPLACE TEMP VIEW t AS SELECT * FROM VALUES (1, 1, 10), (1, 2, 20)
> AS t(id, k, v);
> SELECT * FROM t PIVOT (sum(v) FOR k IN (1 AS x, 1 AS y));
> java.lang.AssertionError: index (1) should < 1
> at
> org.apache.spark.sql.catalyst.expressions.UnsafeRow.assertIndexIsValid(UnsafeRow.java:126)
> at
> org.apache.spark.sql.catalyst.expressions.UnsafeRow.setLong(UnsafeRow.java:224)
> at
> org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.update(PivotFirst.scala)
>
> {code}
>
> The standard (non-PivotFirst) path handles the same query correctly, which
> gives
> the expected answer, one output column per listed pivot value:
>
> {code:java}
> SELECT * FROM t PIVOT (collect_list(v) FOR k IN (1 AS x, 1 AS y));
> --> [1, [10], [10]]{code}
>
> Any pair of pivot values that the lookup treats as equal triggers it, e.g.
> IN (1 AS x, 1 AS y), IN ('a' AS x, 'a' AS y), IN (0.0D, -0.0D), and -- for
> collated string columns -- IN ('a', 'A') under UTF8_LCASE.
> unit tests run with -ea so this surfaces as an AssertionError, but in
> production assertIndexIsValid is a no-op, making this an unchecked
> out-of-bounds
> write into the Tungsten aggregation buffer page rather than a clean failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
