[ https://issues.apache.org/jira/browse/SPARK-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Armbrust resolved SPARK-11894. -------------------------------------- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9904 [https://github.com/apache/spark/pull/9904] > Incorrect results are returned when using null > ---------------------------------------------- > > Key: SPARK-11894 > URL: https://issues.apache.org/jira/browse/SPARK-11894 > Project: Spark > Issue Type: Sub-task > Components: SQL > Affects Versions: 1.6.0 > Reporter: Xiao Li > Fix For: 1.6.0 > > > In DataSet APIs, the following two datasets are the same. > Seq((new java.lang.Integer(0), "1"), (new java.lang.Integer(22), > "2")).toDS() > Seq((null.asInstanceOf[java.lang.Integer],, "1"), (new > java.lang.Integer(22), "2")).toDS() > Note: java.lang.Integer is Nullable. > It could generate an incorrect result. For example, > val ds1 = Seq((null.asInstanceOf[java.lang.Integer], "1"), (new > java.lang.Integer(22), "2")).toDS() > val ds2 = Seq((null.asInstanceOf[java.lang.Integer], "1"), (new > java.lang.Integer(22), "2")).toDS()//toDF("key", "value").as('df2) > val res1 = ds1.joinWith(ds2, lit(true)).collect() > The expected result should be > ((null,1),(null,1)) > ((22,2),(null,1)) > ((null,1),(22,2)) > ((22,2),(22,2)) > The actual result is > ((0,1),(0,1)) > ((22,2),(0,1)) > ((0,1),(22,2)) > ((22,2),(22,2)) -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org