DarkWanderer commented on PR #23851: URL: https://github.com/apache/datafusion/pull/23851#issuecomment-6588081116
Sure. The optimization specifically does the following: * Checks if any of the filters are of matching shape - `=`, `NOT IN`, `IN` * For these filters, checks statistics to see if row groups for this column are "pure dictionary" (not dictionary+`PLAIN` fallback) * If there are, fetch dictionaries for this column and use them to prune row groups where criterion is unsatisfiable This is geared toward needle-in-haystack queries in data hosted on S3, with the following idea: * If you have 1000 row groups and 5% FP Bloom Filter, you will either incur 50x billed read operations, or 20x read amplification (if whole file is read) * With this optimization, only needed row group is read The benchmark run did not show improvement because the option is opt-in - because in some cases it produces small regression due to dictionary check overhead. I leave it up to you to judge usefulness of this option @alamb @adriangb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
