alamb commented on issue #5357: URL: https://github.com/apache/arrow-datafusion/issues/5357#issuecomment-1441721936
@avantgardnerio I wonder if your `TableProvider` could return `TableProviderFilterPushDown::Inexact` for all the filters * DataFusion will not rely on pushed down filters for correctness (there will still be a Filter after the scan) * The `TableProvider` can then figure out what, if any, indexes can be used given the list of predicates it gets. * Any predicate that can't be used in the index is simply ignored https://docs.rs/datafusion/18.0.0/datafusion/datasource/datasource/enum.TableProviderFilterPushDown.html#variant.Inexact Not only does this seem (?) to handle the usecase in this ticket's description, it would also allow you to handle predicates where you only have indexes on part of the predicate. For example, instead of ```sql WHERE s_i_id = $1 AND s_w_id = $2 ``` What if you had a predicate like ```sql WHERE s_i_id = $1 AND s_w_id = $2 AND s_i_age > 21 ``` Or something? You still probably want to use the multi-column index on `(s_i_id, s_w_id)` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
