Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19494#discussion_r145637774
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala
---
@@ -102,6 +102,7 @@ case class InMemoryTableScanExec(
case IsNull(a: Attribute) => statsFor(a).nullCount > 0
case IsNotNull(a: Attribute) => statsFor(a).count -
statsFor(a).nullCount > 0
+ case In(_: AttributeReference, list: Seq[Expression]) if list.isEmpty
=> Literal.FalseLiteral
--- End diff --
Under this special context of filtering partitions, this `In` with empty
list will result in a false literal in the end, no matter the attribute is null
or not. We don't possibly have some expressions like `IsNull(In(a, Nil))` as
the filter predicate for now.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
