shrirangmhalgi commented on code in PR #59004:
URL: https://github.com/apache/spark/pull/59004#discussion_r4089029081
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala:
##########
@@ -72,6 +72,27 @@ case class Person(name: String, age: Int, contacts:
Seq[Contact])
abstract class OrcQueryTest extends OrcTest {
import testImplicits._
+ test("SPARK-59605: ORC IN filter with NULL matches with pushdown on and
off") {
Review Comment:
Consider adding a NOT IN case - it's the subtle one under three-valued logic:
```
// NOT IN with NULL: under three-valued logic, NOT IN (1, 3, NULL) returns
// 0 rows because every non-match evaluates to NULL (not TRUE).
Seq("true", "false").foreach { pushdown =>
withSQLConf(SQLConf.ORC_FILTER_PUSHDOWN_ENABLED.key -> pushdown) {
checkAnswer(
spark.read.orc(path)
.where("id NOT IN (1, 3, NULL)")
.select("id", "label"),
Seq.empty[Row])
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
