I agree this should work. We just haven't finished killing the old reflection based conversion logic now that we have more powerful/efficient encoders. Please open a JIRA.
On Sun, Sep 25, 2016 at 2:41 PM, Koert Kuipers <ko...@tresata.com> wrote: > after having gotten used to have case classes represent complex structures > in Datasets, i am surprised to find out that when i work in DataFrames with > udfs no such magic exists, and i have to fall back to manipulating Row > objects, which is error prone and somewhat ugly. > > for example: > case class Person(name: String, age: Int) > > val df = Seq((Person("john", 33), 5), (Person("mike", 30), > 6)).toDF("person", "id") > val df1 = df.withColumn("person", udf({ (p: Person) => p.copy(age = p.age > + 1) }).apply(col("person"))) > df1.printSchema > df1.show > > leads to: > java.lang.ClassCastException: > org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema > cannot be cast to Person >