one of our unit tests broke with changes in spark 2.0 snapshot in last few
days (or maybe i simple missed it longer). i think it boils down to this:
val df1 = sc.makeRDD(1 to 3).toDF
val df2 = df1.map(row => Row(row(0).asInstanceOf[Int] +
1))(RowEncoder(df1.schema))
println(s"schema before ${df1.schema} and after ${df2.schema}")
i get:
schema before StructType(StructField(value,IntegerType,false)) and after
StructType(StructField(value,IntegerType,true))
it is the change in nullability that i did not expect.
