[
https://issues.apache.org/jira/browse/SPARK-11390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14995246#comment-14995246
]
Vishesh Garg commented on SPARK-11390:
--------------------------------------
So if I understand it correctly, setting of the filterPushdown flag is
immaterial, since what happens under the hood remains unaffected. If that is
the case then the present print of the plan tree is correct and needs no
amendment. Merging the nodes would send out the wrong indication that the
filter is actually getting pushed.
> Query plan with/without filterPushdown indistinguishable
> --------------------------------------------------------
>
> Key: SPARK-11390
> URL: https://issues.apache.org/jira/browse/SPARK-11390
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.5.1
> Environment: All
> Reporter: Vishesh Garg
> Priority: Minor
>
> The execution plan of a query remains the same regardless of whether the
> filterPushdown flag has been set to "true" or "false", as can be seen below:
> ======
> scala> sqlContext.setConf("spark.sql.orc.filterPushdown", "false")
> scala> sqlContext.sql("SELECT name FROM people WHERE age = 15").explain()
> == Physical Plan ==
> Project [name#6]
> Filter (age#7 = 15)
> Scan OrcRelation[hdfs://localhost:9000/user/spec/people][name#6,age#7]
> scala> sqlContext.setConf("spark.sql.orc.filterPushdown", "true")
> scala> sqlContext.sql("SELECT name FROM people WHERE age = 15").explain()
> == Physical Plan ==
> Project [name#6]
> Filter (age#7 = 15)
> Scan OrcRelation[hdfs://localhost:9000/user/spec/people][name#6,age#7]
> ======
> Ideally, when the filterPushdown flag is set to "true", both the scan and the
> filter nodes should be merged together to make it clear that the filtering is
> being done by the data source itself.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
