gatorsmile commented on issue #23315: [SPARK-26366][SQL] ReplaceExceptWithFilter should consider NULL as False URL: https://github.com/apache/spark/pull/23315#issuecomment-447942170 @mgaido91 Could you add an end-to-end test case with all the enumerated cases? Below is just an example. ```Scala test("SPARK-26366: verify ReplaceExceptWithFilter") { Seq(true, false).foreach { enabled => withSQLConf(SQLConf.REPLACE_EXCEPT_WITH_FILTER.key -> enabled.toString) { withTable("tab1") { // TODO: use DF APIs to make it shorter spark.sql( """ |CREATE TABLE tab1 (col1 INT, col2 INT, col3 INT) using PARQUET """.stripMargin) spark.sql("INSERT INTO tab1 VALUES (0, 3, 5)") spark.sql("INSERT INTO tab1 VALUES (0, 3, NULL)") spark.sql("INSERT INTO tab1 VALUES (NULL, 3, 5)") spark.sql("INSERT INTO tab1 VALUES (0, NULL, 5)") spark.sql("INSERT INTO tab1 VALUES (0, NULL, NULL)") spark.sql("INSERT INTO tab1 VALUES (NULL, NULL, 5)") spark.sql("INSERT INTO tab1 VALUES (NULL, 3, NULL)") spark.sql("INSERT INTO tab1 VALUES (NULL, NULL, NULL)") val df = spark.read.table("tab1") // TODO: add more conditions val where = """ |(col1 IS NULL AND col2 >= 3) |OR (col1 IS NOT NULL AND col2 >= 0) """.stripMargin val df1 = df.filter(where) val df2 = df.except(df_a) val df3 = df.except(df_b) // TODO: compare df1 and df3 and check if they are the same } } } } ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
