[
https://issues.apache.org/jira/browse/SPARK-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272074#comment-15272074
]
Cheng Lian commented on SPARK-14139:
------------------------------------
As discussed in SPARK-15112. I think we should revert to use encoder schema as
Dataset schema. But it involves a little bit more work than just changing the
schema to get everything right. I'm looking into these related issues. Thanks
for reporting and your initial investigation!
> Dataset loses nullability in operations with RowEncoder
> -------------------------------------------------------
>
> Key: SPARK-14139
> URL: https://issues.apache.org/jira/browse/SPARK-14139
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: koert kuipers
> Priority: Minor
>
> When i do
> {noformat}
> val df1 = sc.makeRDD(1 to 3).toDF
> val df2 = df1.map(row => Row(row(0).asInstanceOf[Int] +
> 1))(RowEncoder(df1.schema))
> println(s"schema before ${df1.schema} and after ${df2.schema}")
> {noformat}
> I get:
> {noformat}
> schema before StructType(StructField(value,IntegerType,false)) and after
> StructType(StructField(value,IntegerType,true))
> {noformat}
> The change in field nullable is unexpected and i consider it a bug.
> This bug was introduced in:
> [SPARK-13244][SQL] Migrates DataFrame to Dataset
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
