[
https://issues.apache.org/jira/browse/ARROW-13498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17390754#comment-17390754
]
Ben Kietzman commented on ARROW-13498:
--------------------------------------
This is as-designed: the ScanNode only uses the filter for predicate pushdown-
so in the context of datasets, you would exclude some files whose partition
expression contradicted the filter. It does not remove all rows excluded by the
filter- that is deferred to FilterNode.
Similarly, ScanNode does not perform projection- it ensures that fields
referenced by the projection are loaded from disk (which for example in the
case of parquet means you will pay less IO if you reference fewer fields).
> [C++] ScanNode takes filter but doesn't filter
> ----------------------------------------------
>
> Key: ARROW-13498
> URL: https://issues.apache.org/jira/browse/ARROW-13498
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
> Reporter: Neal Richardson
> Priority: Major
> Fix For: 6.0.0
>
>
> This turned out to be my confusion in ARROW-13344: because the ScanNode takes
> a filter and projection, I wasn't creating a FilterNode because I assume that
> the filter is already applied--why else would I provide a filter to the
> ScanNode? But it turns out that if you don't Filter again, you get unfiltered
> results:
> {code}
> Table$create(
> group=c(1, 2),
> value=c(5, 6)
> ) %>%
> filter(value > 5) %>%
> group_by(group) %>%
> summarize(sum(value)) %>%
> collect()
> # A tibble: 2 x 2
> group `sum(value)`
> <dbl> <dbl>
> 1 1 5
> 2 2 6
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
