Eric Yang created SPARK-59684:
---------------------------------
Summary: pivot() on a struct column fails unless the pivot values
are given explicitly
Key: SPARK-59684
URL: https://issues.apache.org/jira/browse/SPARK-59684
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Eric Yang
{{pivot()}} on a struct column fails when Spark collects the distinct values
itself:
{code:java}
scala> Seq(1.0d).toDF("v").selectExpr("v", "struct(v, v) AS
s").groupBy("v").pivot("s").count()
org.apache.spark.SparkRuntimeException: [UNSUPPORTED_FEATURE.PIVOT_TYPE] The
feature is not supported:
Pivoting by the value '[1.0,1.0]' of the column data type "STRUCT<v: DOUBLE
NOT NULL, v: DOUBLE NOT NULL>".
{code}
This happens for any struct column, regardless of the field types or their
nullability. Passing the same values explicitly as columns works:
{code:java}
df.groupBy("v").pivot($"s", Seq(struct(lit(1.0d), lit(1.0d)))).count()
{code}
so only the {{pivot(pivotColumn)}} overload is affected. Pivoting by an array
column works in both forms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
