ok will create jira
On Mon, Sep 26, 2016 at 3:27 PM, Michael Armbrust <mich...@databricks.com>
wrote:
> I agree this should work. We just haven't finished killing the old
> reflection based conversion logic now that we have more powerful/efficient
> encoders. Please open a JIRA.
>
> On Sun, Sep 25, 2016 at 2:41 PM, Koert Kuipers <ko...@tresata.com> wrote:
>
>> after having gotten used to have case classes represent complex
>> structures in Datasets, i am surprised to find out that when i work in
>> DataFrames with udfs no such magic exists, and i have to fall back to
>> manipulating Row objects, which is error prone and somewhat ugly.
>>
>> for example:
>> case class Person(name: String, age: Int)
>>
>> val df = Seq((Person("john", 33), 5), (Person("mike", 30),
>> 6)).toDF("person", "id")
>> val df1 = df.withColumn("person", udf({ (p: Person) => p.copy(age = p.age
>> + 1) }).apply(col("person")))
>> df1.printSchema
>> df1.show
>>
>> leads to:
>> java.lang.ClassCastException: org.apache.spark.sql.catalyst.
>> expressions.GenericRowWithSchema cannot be cast to Person
>>
>
>
