Eric Yang created SPARK-58959:
---------------------------------
Summary: Duplicate PIVOT values corrupt the aggregation buffer on
the PivotFirst fast path
Key: SPARK-58959
URL: https://issues.apache.org/jira/browse/SPARK-58959
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Eric Yang
When a PIVOT ... IN list contains two values that compare equal, the optimized
PivotFirst fast path writes outside the slots it owns in the aggregation buffer.
Repro:
{code:java}
CREATE OR REPLACE TEMP VIEW t AS SELECT * FROM VALUES (1, 1, 10), (1, 2, 20) AS
t(id, k, v);
SELECT * FROM t PIVOT (sum(v) FOR k IN (1 AS x, 1 AS y));
java.lang.AssertionError: index (1) should < 1
at
org.apache.spark.sql.catalyst.expressions.UnsafeRow.assertIndexIsValid(UnsafeRow.java:126)
at
org.apache.spark.sql.catalyst.expressions.UnsafeRow.setLong(UnsafeRow.java:224)
at
org.apache.spark.sql.catalyst.expressions.aggregate.PivotFirst.update(PivotFirst.scala)
{code}
The standard (non-PivotFirst) path handles the same query correctly, which gives
the expected answer, one output column per listed pivot value:
{code:java}
SELECT * FROM t PIVOT (collect_list(v) FOR k IN (1 AS x, 1 AS y));
--> [1, [10], [10]]{code}
Any pair of pivot values that the lookup treats as equal triggers it, e.g.
IN (1 AS x, 1 AS y), IN ('a' AS x, 'a' AS y), IN (0.0D, -0.0D), and -- for
collated string columns -- IN ('a', 'A') under UTF8_LCASE.
unit tests run with -ea so this surfaces as an AssertionError, but in
production assertIndexIsValid is a no-op, making this an unchecked out-of-bounds
write into the Tungsten aggregation buffer page rather than a clean failure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
