Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/19494#discussion_r144621674
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala
---
@@ -104,7 +104,8 @@ case class InMemoryTableScanExec(
case In(a: AttributeReference, list: Seq[Expression]) if
list.forall(_.isInstanceOf[Literal]) =>
list.map(l => statsFor(a).lowerBound <= l.asInstanceOf[Literal] &&
- l.asInstanceOf[Literal] <= statsFor(a).upperBound).reduce(_ || _)
+ l.asInstanceOf[Literal] <= statsFor(a).upperBound)
--- End diff --
This still looks more complex and less efficient than
```
list.exists(l => statsFor(a).lowerBound <= l.asInstanceOf[Literal] &&
l.asInstanceOf[Literal] <= statsFor(a).upperBound)
```
or better
```
val stats = statsFor(a)
list.exists { l =>
val literal = l.asInstanceOf[Literal]
stats.lowerBound <= literal && literal <= stats.upperBound
}
```
The point being that you should be able to short-circuit evaluation here.
Or have I missed something basic like that these aren't Booleans?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
