Henrique dos Santos Goulart created SPARK-23273:
---------------------------------------------------
Summary: Spark Dataset withColumn - schema column order isn't the
same as case class paramether order
Key: SPARK-23273
URL: https://issues.apache.org/jira/browse/SPARK-23273
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.2.1
Reporter: Henrique dos Santos Goulart
{code}
case class OnlyAge(age: Int)
case class NameAge(name: String, age: Int)
val ds1 = spark.emptyDataset[NameAge]
val ds2 = spark
.createDataset(Seq(OnlyAge(1)))
.withColumn("name", lit("henriquedsg89"))
.as[NameAge]
ds1.show()
ds2.show()
ds1.union(ds2)
{code}
It's going to raise this error:
{noformat}
Cannot up cast `age` from string to int as it may truncate
The type path of the target object is:
- field (class: "scala.Int", name: "age")
- root class: "dw.NameAge"{noformat}
It seems that .as[CaseClass] doesn't keep the order of paramethers that is
typed on case class.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
