[ https://issues.apache.org/jira/browse/SPARK-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347704#comment-15347704 ]
Apache Spark commented on SPARK-16181: -------------------------------------- User 'cloud-fan' has created a pull request for this issue: https://github.com/apache/spark/pull/13884 > Incorrect behavior for isNull filter > ------------------------------------ > > Key: SPARK-16181 > URL: https://issues.apache.org/jira/browse/SPARK-16181 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.0.0 > Reporter: Kevin Chen > > Repro: > JavaRDD<Row> leftRdd = > javaSparkContext.parallelize(ImmutableList.of(RowFactory.create("x"))); > JavaRDD<Row> rightRdd = > javaSparkContext.parallelize(ImmutableList.of(RowFactory.create("y"))); > StructType schema = DataTypes.createStructType(ImmutableList.of( > DataTypes.createStructField("col", DataTypes.StringType, > true))); > Dataset<Row> left = sparkSession.createDataFrame(leftRdd, schema); > Dataset<Row> right = sparkSession.createDataFrame(rightRdd, schema); > // add a column to the right > Dataset<Row> withConstantColumn = right.withColumn("new", > functions.lit(true)); > // do a left join. Nothing matches; expect Dataset joined to have a > single row ['x', null, null] > Column joinCondition = left.col("col").equalTo(right.col("col")); > Dataset<Row> joined = left.join(withConstantColumn, joinCondition, > LeftOuter.toString()); > // filter for nulls, still expect the single row ['x', null, null] > Dataset<Row> filtered = joined.filter(functions.col("new").isNull()); > // This fails with 1 != 0 > Assert.assertEquals(1, filtered.count()); > [~rxin] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org