Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21882#discussion_r205504149
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryTableScanExec.scala
---
@@ -183,6 +183,13 @@ case class InMemoryTableScanExec(
private val stats = relation.partitionStatistics
private def statsFor(a: Attribute) = stats.forAttribute(a)
+ // For some ColumnStats, for instance, ObjectColumnStats always has
nulls for lower and upper
+ // bounds.
+ private def nullSafeEval(
+ attr: AttributeReference)(func: AttributeReference => Expression):
Expression = {
+ attr.isNull || func(attr)
--- End diff --
this adds extra runtime null check and may introduce perf regression. How
about we follow the hive partition pruning and only create filters for
non-complex type? e.g.
```
object ExtractableLiteral {
def unapply(expr: Expression): Option[Expression] = {
if (expr.dataType.isInstanceOf[AtomicType]) Some(expr) else None
}
}
...
case EqualTo(a: AttributeReference, ExtractableLiteral(l)) =>
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
