EnricoMi commented on code in PR #36150:
URL: https://github.com/apache/spark/pull/36150#discussion_r881793259
##########
sql/core/src/test/scala/org/apache/spark/sql/MeltSuite.scala:
##########
@@ -0,0 +1,297 @@
+/*
Review Comment:
When I run above example, I get this error. Looks like resolving the columns
does not work with the pivoted Dataset.
```
Resolved attribute(s) dotNET#206,Java#207 missing from
year#50,dotNET#224,Java#225 in operator !Expand [[year#50, dotNET,
cast(dotNET#206 as double)], [year#50, Java, cast(Java#207 as double)]],
[year#50, variable#215, value#216]. Attribute(s) with the same name appear in
the operation: dotNET,Java. Please check if the right attribute(s) are used.;
!Expand [[year#50, dotNET, cast(dotNET#206 as double)], [year#50, Java,
cast(Java#207 as double)]], [year#50, variable#215, value#216]
+- Project [year#50, __pivot_sum(earnings) AS `sum(earnings)`#223[0] AS
dotNET#224, __pivot_sum(earnings) AS `sum(earnings)`#223[1] AS Java#225]
+- Aggregate [year#50], [year#50, pivotfirst(course#49,
sum(earnings)#217, dotNET, Java, 0, 0) AS __pivot_sum(earnings) AS
`sum(earnings)`#223]
+- Aggregate [year#50, course#49], [year#50, course#49,
sum(earnings#51) AS sum(earnings)#217]
+- SerializeFromObject [staticinvoke(class
org.apache.spark.unsafe.types.UTF8String, StringType, fromString,
knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).course, true, false,
true) AS course#49, knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).year AS year#50,
knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).earnings AS
earnings#51]
+- ExternalRDD [obj#48]
org.apache.spark.sql.AnalysisException: Resolved attribute(s)
dotNET#206,Java#207 missing from year#50,dotNET#224,Java#225 in operator
!Expand [[year#50, dotNET, cast(dotNET#206 as double)], [year#50, Java,
cast(Java#207 as double)]], [year#50, variable#215, value#216]. Attribute(s)
with the same name appear in the operation: dotNET,Java. Please check if the
right attribute(s) are used.;
!Expand [[year#50, dotNET, cast(dotNET#206 as double)], [year#50, Java,
cast(Java#207 as double)]], [year#50, variable#215, value#216]
+- Project [year#50, __pivot_sum(earnings) AS `sum(earnings)`#223[0] AS
dotNET#224, __pivot_sum(earnings) AS `sum(earnings)`#223[1] AS Java#225]
+- Aggregate [year#50], [year#50, pivotfirst(course#49,
sum(earnings)#217, dotNET, Java, 0, 0) AS __pivot_sum(earnings) AS
`sum(earnings)`#223]
+- Aggregate [year#50, course#49], [year#50, course#49,
sum(earnings#51) AS sum(earnings)#217]
+- SerializeFromObject [staticinvoke(class
org.apache.spark.unsafe.types.UTF8String, StringType, fromString,
knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).course, true, false,
true) AS course#49, knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).year AS year#50,
knownnotnull(assertnotnull(input[0,
org.apache.spark.sql.test.SQLTestData$CourseSales, true])).earnings AS
earnings#51]
+- ExternalRDD [obj#48]
```
When I checkpoint the pivoted Dataset (so that analysis is stopped), melting
works just fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
