zhouhuazheng created SPARK-29626: ------------------------------------ Summary: notEqual() should return true when the one is null, the other is not null Key: SPARK-29626 URL: https://issues.apache.org/jira/browse/SPARK-29626 Project: Spark Issue Type: Improvement Components: Documentation Affects Versions: 2.4.4 Reporter: zhouhuazheng
the one is null,the other is not null, then use the function notEqual(), we hope it return true . eg: scala> df.show() +------+-------+ | age| name| +------+-------+ | null|Michael| | 30| Andy| | 19| Justin| | 35| null| | 19| Justin| | null| null| |Justin| Justin| | 19| 19| +------+-------+ scala> df.filter(col("age").notEqual(col("name"))).show +---+------+ |age| name| +---+------+ | 30| Andy| | 19|Justin| | 19|Justin| +---+------+ scala> df.filter(col("age").equalTo(col("name"))).show +------+------+ | age| name| +------+------+ | null| null| |Justin|Justin| | 19| 19| +------+------+ -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org