e-dard opened a new issue #408:
URL: https://github.com/apache/arrow-datafusion/issues/408
**Describe the bug**
If you run a `LogicalPlan` that has a `TableProviderFilterPushDown`
`TableProvider`, then the filter pushdown optimiser rule will keep adding
filters to the pushed down table scan without checking if they're already
present.
Given the plan:
```
Filter: #a Eq Int64(1)
TableScan: projection=None
```
it will be optimised to the following plan the first time it runs through
the filter:
```
Filter: #a Eq Int64(1)
TableScan: projection=None, filters=[#a Eq Int64(1)]
```
If you run it through the optimiser again then it will not be left as is.
Instead it will become:
```
Filter: #a Eq Int64(1)
TableScan: projection=None, filters=[#a Eq Int64(1), #a Eq Int64(1)]
```
**To Reproduce**
Run a logical plan that pushes predicates down to the table provider through
the plan optimiser twice. All pushed down expression filters will be duplicated
in the table scan filters.
**Expected behavior**
The plan should not change if it's ran through the optimiser multiple times.
**Additional context**
Whilst the current behaviour produces correct plans they are not optimal
because they can contain redunant duplicate filters if they're run through the
optimiser multiple times.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
