Can you see if this is the patch that caused the issue?
https://github.com/apache/spark/pull/11737
On Wed, Mar 23, 2016 at 2:20 PM, Koert Kuipers <ko...@tresata.com> wrote:
> one of our unit tests broke with changes in spark 2.0 snapshot in last few
> days (or maybe i simple missed it longer). i think it boils down to this:
>
> val df1 = sc.makeRDD(1 to 3).toDF
> val df2 = df1.map(row => Row(row(0).asInstanceOf[Int] +
> 1))(RowEncoder(df1.schema))
> println(s"schema before ${df1.schema} and after ${df2.schema}")
>
> i get:
> schema before StructType(StructField(value,IntegerType,false)) and after
> StructType(StructField(value,IntegerType,true))
>
> it is the change in nullability that i did not expect.
>
