[
https://issues.apache.org/jira/browse/SPARK-59678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated SPARK-59678:
------------------------------
Description:
{{pivot()}} on an array column fails at analysis with an error that prints the
same type on both sides:
{code:java}
scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS
a").groupBy("v").pivot("a").count()
org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH]
Invalid pivot value '[1.0]':
value data type array<double> does not match pivot column data type
array<double>
{code}
It only happens when the array's element type is non-nullable; the same query
succeeds with {{{}Seq(Some(1.0d), None){}}}. There is no workaround, since the
pivot values are collected by Spark rather than supplied by the user.
was:
{{pivot()}} on an array column fails at analysis with an error that prints the
same type on both sides:
{code}
scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS
a").groupBy("v").pivot("a").count()
org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH]
Invalid pivot value '[1.0]':
value data type array<double> does not match pivot column data type
array<double>
{code}
It only happens when the array's element type is non-nullable; the same query
succeeds with {{Seq(Some(1.0d), None)}}. There is no workaround, since the
pivot values are collected by Spark rather than supplied by the user.
> pivot() on an array column fails when the column's element type is
> non-nullable
> -------------------------------------------------------------------------------
>
> Key: SPARK-59678
> URL: https://issues.apache.org/jira/browse/SPARK-59678
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Eric Yang
> Priority: Major
>
> {{pivot()}} on an array column fails at analysis with an error that prints
> the same type on both sides:
> {code:java}
> scala> Seq(1.0d).toDF("v").selectExpr("v", "array(v) AS
> a").groupBy("v").pivot("a").count()
> org.apache.spark.sql.AnalysisException: [PIVOT_VALUE_DATA_TYPE_MISMATCH]
> Invalid pivot value '[1.0]':
> value data type array<double> does not match pivot column data type
> array<double>
> {code}
> It only happens when the array's element type is non-nullable; the same query
> succeeds with {{{}Seq(Some(1.0d), None){}}}. There is no workaround, since
> the pivot values are collected by Spark rather than supplied by the user.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
