avantgardnerio opened a new issue, #5357:
URL: https://github.com/apache/arrow-datafusion/issues/5357
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
I have a `TableProvider` that supports multi-column indexes and could
theoretically support filter pushdown, however when running the query:
```
SELECT s_quantity, s_data, s_dist_01, s_dist_02, s_dist_03, s_dist_04,
s_dist_05, s_dist_06, s_dist_07, s_dist_08, s_dist_09, s_dist_10 FROM stock
WHERE s_i_id = $1 AND s_w_id = $2 FOR UPDATE
```
on the table
```
create table stock (
s_i_id int not null,
s_w_id smallint not null,
...
s_data varchar(50),
PRIMARY KEY(s_w_id, s_i_id) ) ;
```
I observe the following in my logs:
```
ArrowTableProvider::supports_filter_pushdown() proj=stock.s_i_id =
Int32(3604)
ArrowTableProvider::supports_filter_pushdown() proj=stock.s_w_id = Int32(1)
ArrowTableProvider::supports_filter_pushdown() proj=stock.s_i_id =
Int32(3604)
ArrowTableProvider::supports_filter_pushdown() proj=stock.s_w_id = Int32(1)
```
```
fn supports_filter_pushdown(&self, filter: &Expr) ->
Result<TableProviderFilterPushDown, DataFusionError> {
// TODO: parse expression and return "true" if it's a simple range
on an index we have
println!("ArrowTableProvider::supports_filter_pushdown()
filter={filter:?}");
Ok(TableProviderFilterPushDown::Unsupported)
}
```
So I would have to do complicated logic (append each portion of the
predicate, always return "true", then "and" them back together during scan())
to properly apply this filter push down in my code.
**Describe the solution you'd like**
DataFusion would provide raw `Expr`s of the entire filter it is trying to
push down, or - as a compromise, try first with the whole filter, then if the
`TableProvider` returns `Unsupported` try again with each part as suggested by
@andygrove .
**Describe alternatives you've considered**
- Try to reconstruct the original filter in my TableProvider asynchronously
- Give up and don't use indexes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
