[
https://issues.apache.org/jira/browse/ARROW-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neville Dipale updated ARROW-10547:
-----------------------------------
Component/s: Rust - DataFusion
> [Rust][DataFusion] Filter pushdown loses filters if below a user defined node
> -----------------------------------------------------------------------------
>
> Key: ARROW-10547
> URL: https://issues.apache.org/jira/browse/ARROW-10547
> Project: Apache Arrow
> Issue Type: Bug
> Components: Rust - DataFusion
> Reporter: Andrew Lamb
> Assignee: Andrew Lamb
> Priority: Major
>
> I have a LogicalPlan like this (where "Extension" is some extension node
> type):
> {code}
> Extension [non_null_column:Utf8]
> Filter: #state Eq Utf8("MA") [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
> InMemoryScan: projection=None [city:Utf8;N, state:Utf8;N,
> borough:Utf8;N]
> {code}
> When I run this plan through {{ExecutionContext::optimize}} the plan that
> results is as follows (note the filter has been lost):
> {code}
> Extension [non_null_column:Utf8]
> InMemoryScan: projection=Some([0, 1, 2]) [city:Utf8;N, state:Utf8;N,
> borough:Utf8;N]
> {code}
> I have debugged the problem and the root cause of the issue is that the
> FilterPushdown logic is not recursing into the inputs of user defined nodes.
> I will get a PR up showing and fixing the problem.
> More generally, I am not sure how / if the filter pushdown logic would work
> if there are multiple child inputs as no existing LogicalPlan variant his
> multiple inputs at this time. We will have to handle this when / if we want
> to support joins (typically filters get pushed down each join input if
> possible)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
