[jira] [Assigned] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs
[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17668: Assignee: Apache Spark > Support representing structs with case classes and tuples in spark sql udf > inputs > - > > Key: SPARK-17668 > URL: https://issues.apache.org/jira/browse/SPARK-17668 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 2.0.0 >Reporter: koert kuipers >Assignee: Apache Spark >Priority: Minor > > after having gotten used to have case classes represent complex structures in > Datasets, i am surprised to find out that when i work in DataFrames with udfs > no such magic exists, and i have to fall back to manipulating Row objects, > which is error prone and somewhat ugly. > for example: > {noformat} > case class Person(name: String, age: Int) > val df = Seq((Person("john", 33), 5), (Person("mike", 30), 6)).toDF("person", > "id") > val df1 = df.withColumn("person", udf({ (p: Person) => p.copy(age = p.age + > 1) }).apply(col("person"))) > df1.printSchema > df1.show > {noformat} > leads to: > {noformat} > java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast > to Person > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-17668) Support representing structs with case classes and tuples in spark sql udf inputs
[ https://issues.apache.org/jira/browse/SPARK-17668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-17668: Assignee: (was: Apache Spark) > Support representing structs with case classes and tuples in spark sql udf > inputs > - > > Key: SPARK-17668 > URL: https://issues.apache.org/jira/browse/SPARK-17668 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 2.0.0 >Reporter: koert kuipers >Priority: Minor > > after having gotten used to have case classes represent complex structures in > Datasets, i am surprised to find out that when i work in DataFrames with udfs > no such magic exists, and i have to fall back to manipulating Row objects, > which is error prone and somewhat ugly. > for example: > {noformat} > case class Person(name: String, age: Int) > val df = Seq((Person("john", 33), 5), (Person("mike", 30), 6)).toDF("person", > "id") > val df1 = df.withColumn("person", udf({ (p: Person) => p.copy(age = p.age + > 1) }).apply(col("person"))) > df1.printSchema > df1.show > {noformat} > leads to: > {noformat} > java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast > to Person > {noformat} -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org