Jefffrey commented on issue #9405:
URL:
https://github.com/apache/arrow-datafusion/issues/9405#issuecomment-1973997314
It seems it is because of this in your code:
```rust
fn supports_filters_pushdown(
&self,
_filters: &[&datafusion::prelude::Expr],
) ->
datafusion::error::Result<Vec<datafusion::logical_expr::TableProviderFilterPushDown>>
{
Ok(vec![TableProviderFilterPushDown::Unsupported])
}
```
It is actually important to consider the `_filters` argument as the function
should return for each filter, whether it supports being pushed down or not.
See the default implementation:
https://github.com/apache/arrow-datafusion/blob/2a490e48c924b76a2d389227c674b9f81b0404c8/datafusion/expr/src/table_source.rs#L89-L100
Because you returned a Vec with only a single element, later in the
push_down_filter optimizer, when it is checking each filter, it seems it won't
push down the first filter (`id > 1`) but pushes down the second (`id < 3`):
https://github.com/apache/arrow-datafusion/blob/2a490e48c924b76a2d389227c674b9f81b0404c8/datafusion/optimizer/src/push_down_filter.rs#L857-L865
- Since zip will only be as long as the shortest
Pushing down the second kinda pushes it into the void hence only `id > 1`
took effect.
I think we need to improve the documentation of
`supports_filters_pushdown(...)` to clarify the expected behaviour, to prevent
footguns like this in the future. Not sure if we can do any checks in the
`push_down_filter` optimizer side, but worth investigating? :thinking:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
