[
https://issues.apache.org/jira/browse/ARROW-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17494780#comment-17494780
]
Weston Pace commented on ARROW-15726:
-------------------------------------
Ok. The projection problem is my fault due to a change I introduced in
https://github.com/apache/arrow/commit/a935c81b595d24179e115d64cda944efa93aa0e0
to make it easier to scan. The change created a default behavior if the user
did not specify a projection or projected_schema. Unfortunately, if the user
supplies a bound projection function but not a projected_schema (which is what
R does) then we could (and were) infer the projected_schema from the arguments
of the projection expression.
The filter problem wasn't actually a problem but an issue with printing. We do
not print the pushdown filter when we print the scan node so I had assumed it
wasn't specified but it was, in fact, there.
> [R] Support push-down projection/filtering in datasets / dplyr
> --------------------------------------------------------------
>
> Key: ARROW-15726
> URL: https://issues.apache.org/jira/browse/ARROW-15726
> Project: Apache Arrow
> Issue Type: Improvement
> Components: R
> Reporter: Weston Pace
> Priority: Major
>
> The following query should read a single column from the target parquet file.
> {noformat}
> open_dataset("lineitem.parquet") %>% select(l_tax) %>% filter(l_tax < 0.01)
> %>% collect()
> {noformat}
> Furthermore, it should apply a pushdown filter to the source node allowing
> parquet row groups to potentially filter out target data.
> At the moment it creates the following exec plan:
> {noformat}
> 3:SinkNode{}
> 2:ProjectNode{projection=[l_tax]}
> 1:FilterNode{filter=(l_tax < 0.01)}
> 0:SourceNode{}
> {noformat}
> There is no projection or filter in the source node. As a result we end up
> reading much more data from disk (the entire file) than we need to (at most a
> single column).
> This _could_ be fixed via heuristics in the dplyr code. However, it may
> quickly get complex (for example, the project comes after the filter, so you
> need to make sure you push down a projection that includes both the columns
> accessed by the filter and the columns accessed by the projection OR can you
> push down the projection through a join [yes you can], how do you know which
> columns to apply to which source node).
> A more complete fix would be to call into some kind of 3rd party optimizer
> (e.g. calcite) after the plan has been created by dplyr but before it is
> passed to the execution engine.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
