Hyukjin Kwon created SPARK-11621:
------------------------------------
Summary: ORC filter pushdown not working properly after new
unhandled filter interface.
Key: SPARK-11621
URL: https://issues.apache.org/jira/browse/SPARK-11621
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.6.0
Reporter: Hyukjin Kwon
After we get the new interface to get rid of filters predicate-push-downed
which are processed in datasource-level
(https://github.com/apache/spark/pull/9399), it dose not push down filters for
ORC.
This is because at {{DataSourceStrategy}}, it is classified to scanning
non-partitioned HadoopFsRelation, and all the filters are treated as unhandled
filters.
Also, since ORC does not support to filter fully record by record but instead
rough results came out, the filters for ORC should not go to unhandled filters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
