cisaacson commented on issue #13994: URL: https://github.com/apache/datafusion/issues/13994#issuecomment-2572223462
FYI I tried a query with a 6 table join and a single binary operation with an equals in the predicate in the `WHERE` clause. For 2 of the tables `supports_pushdown_filters` was definitely called 2 times both with the same `filter`. Could it be related to the optimizer somehow? That is what is called just prior to `supports_pushdown_predicate`. From what I know this fn should be called once per custom data source, with all `filters` for that specific table, otherwise it is very hard to utilize the pushdown functionality (critical to DataFusion and custom data sources). FYI I was able to work around this by evaluating the query at parse time and ignoring the pushdown `filters` in the `scan` after `supports_pushdown_predicates` is called. `scan` is correctly always called once and only once, then I am using an internal evaluation mechanism to return the right results to DataFusion. Everything after that in the process seems to be correct. Any help appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
