Eric Yang created SPARK-59678:
---------------------------------
Summary: pivot() on an array column fails when the column's
element type is non-nullable
Key: SPARK-59678
URL: https://issues.apache.org/jira/browse/SPARK-59678
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Eric Yang
{{pivot()}} on an array column fails at analysis with an error that prints the
same type on both sides:
{code}
scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS
a").groupBy("v").pivot("a").count()
org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH]
Invalid pivot value '[1.0]':
value data type array<double> does not match pivot column data type
array<double>
{code}
It only happens when the array's element type is non-nullable; the same query
succeeds with {{Seq(Some(1.0d), None)}}. There is no workaround, since the
pivot values are collected by Spark rather than supplied by the user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
