adriangb commented on issue #18856: URL: https://github.com/apache/datafusion/issues/18856#issuecomment-5391109904
@jayshrivastava I think I understand a bit of the friction more now. I forget we are doing checks in `execute()` and once I saw that I understood where you are coming from and why adding an `Inexact` option is attractive. We may still end up adding it, but as a first step to make things nicer what do you think of https://github.com/apache/datafusion/pull/24601? Would it at least remove some of the complexity you mention in https://github.com/datafusion-contrib/datafusion-distributed/pull/634? Long term I feel we will make scans actually apply all filters and dissolve the `filter_pushdown` option, which is also needed to keep issues like https://github.com/apache/datafusion/pull/24125 from recurring. At that point scan's won't need to reply `No` and keep a reference for stats pruning, they can always reply `Yes`. Ironically this is blocked because actually evaluating hash join dynamic filters is generally a regression in our standard benchmarks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
