cisaacson commented on issue #13994: URL: https://github.com/apache/datafusion/issues/13994#issuecomment-2573291404
@jonahgao Thanks for explaining this. We can probably work with this but the issue is that since we want some `filters` and not others (in other words some are preferred indexes) we need to keep state between `supports_filters_pushdown` calls. The trait for this fn is `&self` and not `&mut self` so the only way for us to track it is with something like `OnceLock` to maintain the state. I asked about this a while back, if `&mut self` was passed to this fn it would be much easier to control the functionality. One last question: If I have a query with `[filterA,filterB]` and on the initial call I return `[Unsupported,Exact]` and on a subsequent call you send me `[filterA]` and I return `[Unsupported]`, then I believe the `scan` `filters` comes through as `[]` (empty). Is DataFusion still respecting the `Exact` on `filterB`? My guess is not, that is another important problem as if my custom data source only sends back data that matches `filterB` and DataFusion is again filtering all result data by `filterB` again the pushdown is not doing anything. I see this now, it does work because my data source did filter the data properly, but my guess is DataFusion is doing extra work it should not do. For now your explanation gives me a workaround (given the inefficiency I just explained, not a deal breaker but it could be improved). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
