sunchao commented on PR #24526: URL: https://github.com/apache/datafusion/pull/24526#issuecomment-5420207477
> For benches it shows dramatic speed up after values > 20, which is supposedly related to MAX_IN_LIST_SIZE? Yes, as `MAX_IN_LIST_SIZE` grows, the old implementation does more work. For each row group or page: - Old approach: Evaluate a comparison branch for every requested value: “Does this value fall between min and max?” Work grows roughly as `N`. - Compact approach: Sort the values once, then binary-search for the first value ≥ min and check whether it is ≤ max. Work per interval grows roughly as `log₂(N)`. Therefore this is especially useful when we want to push down a large number of sparse IDs or keys to Parquet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
