Andrew Lamb created ARROW-10547:
-----------------------------------
Summary: [Rust][DataFusion] Filter pushdown loses filters if below
a user defined node
Key: ARROW-10547
URL: https://issues.apache.org/jira/browse/ARROW-10547
Project: Apache Arrow
Issue Type: Bug
Reporter: Andrew Lamb
Assignee: Andrew Lamb
I have a LogicalPlan like this (where "Extension" is some extension node type):
{code}
Extension [non_null_column:Utf8]
Filter: #state Eq Utf8("MA") [city:Utf8;N, state:Utf8;N, borough:Utf8;N]
InMemoryScan: projection=None [city:Utf8;N, state:Utf8;N,
borough:Utf8;N]
{code}
When I run this plan through {{ExecutionContext::optimize}} the plan that
results is as follows (note the filter has been lost):
{code}
Extension [non_null_column:Utf8]
InMemoryScan: projection=Some([0, 1, 2]) [city:Utf8;N, state:Utf8;N,
borough:Utf8;N]
{code}
I have debugged the problem and the root cause of the issue is that the
FilterPushdown logic is not recursing into the inputs of user defined nodes. I
will get a PR up showing and fixing the problem.
More generally, I am not sure how / if the filter pushdown logic would work if
there are multiple child inputs as no existing LogicalPlan variant his multiple
inputs at this time. We will have to handle this when / if we want to support
joins (typically filters get pushed down each join input if possible)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
