Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/511#discussion_r12010205
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala ---
@@ -175,12 +175,35 @@ abstract class SparkStrategies extends
QueryPlanner[SparkPlan] {
InsertIntoParquetTable(relation, planLater(child),
overwrite=true)(sparkContext) :: Nil
case logical.InsertIntoTable(table: ParquetRelation, partition,
child, overwrite) =>
InsertIntoParquetTable(table, planLater(child),
overwrite)(sparkContext) :: Nil
- case PhysicalOperation(projectList, filters, relation:
ParquetRelation) =>
- // TODO: Should be pushing down filters as well.
+ case PhysicalOperation(projectList, filters: Seq[Expression],
relation: ParquetRelation) => {
+ val remainingFilters =
+ if
(sparkContext.conf.getBoolean(ParquetFilters.PARQUET_FILTER_PUSHDOWN_ENABLED,
true)) {
+ filters.filter {
+ // Note: filters cannot be pushed down to Parquet if they
contain more complex
+ // expressions than simple "Attribute cmp Literal"
comparisons. Here we remove
+ // all filters that have been pushed down. Note that a
predicate such as
+ // "A AND B" can result in "A" being pushed down.
--- End diff --
However, we will never get `A AND B` right? As I think any conjunctive
predicates will be split by using `PhysicalOperation`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
