[
https://issues.apache.org/jira/browse/SPARK-18861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-18861.
----------------------------------
Resolution: Not A Problem
I see. This seems actually a behaviour documented in
https://github.com/apache/spark/blob/f830bb9170f6b853565d9dd30ca7418b93a54fe3/sql/core/src/main/scala/org/apache/spark/sql/functions.scala#L1196-L1200
The cases except for the first case seem working fine as expected in the
current master
{code}
scala> Seq((1, 2, 3), (2, 3, 4)).toDF("a", "b",
"c").createOrReplaceTempView("t1")
scala> sql("SELECT case when a>b then struct(cast(a as int), cast(b as int))
else struct(cast(c as int), cast(c as int)) end from t1").show()
+-------------------------------------------------------------------------------------------------------------------------------------------------+
|CASE WHEN (a > b) THEN named_struct(col1, CAST(a AS INT), col2, CAST(b AS
INT)) ELSE named_struct(col1, CAST(c AS INT), col2, CAST(c AS INT)) END|
+-------------------------------------------------------------------------------------------------------------------------------------------------+
|
[3,3]|
|
[4,4]|
+-------------------------------------------------------------------------------------------------------------------------------------------------+
{code}
I am resolving this {{Not A Problem}} as the issue seems obsolete to me.
Please reopen this if anyone feels this is an inappropriate action.
> Spark-SQL unconsistent behavior with "struct" expressions
> ---------------------------------------------------------
>
> Key: SPARK-18861
> URL: https://issues.apache.org/jira/browse/SPARK-18861
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Ohad Raviv
>
> We are getting strangly inconsistent behavior with expressions involving
> "struct". Let's start with this simple table:
> {quote}
> Seq((1, 2, 3), (2, 3, 4)).toDF("a", "b", "c").createOrReplaceTempView("t1")
> sql("desc t1").show()
> {quote}
> Then we get this DF:
> {quote}
> |col_name|data_type|comment|
> | a| int| |
> | b| int| |
> | c| int| |
> {quote}
> Now, although we can clearly see that all the fields are of type int, we we
> run:
> {quote}
> sql("SELECT case when a>b then struct(a,b) else struct(c,c) end from t1")
> {quote}
> we get this error:
> {quote}
> org.apache.spark.sql.AnalysisException: cannot resolve 'CASE WHEN (t1.`a` >
> t1.`b`) THEN struct(t1.`a`, t1.`b`) ELSE struct(t1.`c`, t1.`c`) END' due to
> data type mismatch: THEN and ELSE expressions should all be same type or
> coercible to a common type; line 1 pos 7
> {quote}
> if we try this:
> {quote}
> sql("SELECT case when a>b then struct(cast(a as int), cast(b as int)) else
> struct(cast(c as int), cast(c as int)) end from t1")
> {quote}
> we get another exception:
> {quote}
> requirement failed: Unresolved attributes found when constructing
> LocalRelation.
> java.lang.IllegalArgumentException: requirement failed: Unresolved attributes
> found when constructing LocalRelation.
> at scala.Predef$.require(Predef.scala:224)
> at
> org.apache.spark.sql.catalyst.plans.logical.LocalRelation.<init>(LocalRelation.scala:49)
> {quote}
> However, these do work:
> {quote}
> sql("SELECT case when a>b then struct(cast(a as double), cast(b as double))
> else struct(cast(c as double), cast(c as double)) end from t1")
> sql("SELECT case when a>b then struct(cast(a as string), cast(b as string))
> else struct(cast(c as string), cast(c as string)) end from t1")
> {quote}
> any ideas?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
