gruuya commented on issue #24710: URL: https://github.com/apache/datafusion/issues/24710#issuecomment-5666180397
> That PR uses the compact sorted-domain form only for lists strictly larger than MAX_IN_LIST_SIZE (20); Currently `build_predicate_expression` has the following condition guarding the compact representation https://github.com/apache/datafusion/blob/85d4cbb0a9a885f2fd22bfdac468844446ad70d6/datafusion/pruning/src/pruning_predicate.rs#L1777-L1778 However, both of those values resolve to 20, not only `MAX_IN_LIST_SIZE`; the second one defaults to it https://github.com/apache/datafusion/blob/85d4cbb0a9a885f2fd22bfdac468844446ad70d6/datafusion/common/src/config.rs#L1414 Consequently the compact pruning expression never gets built by default, whether for a direct IN clause or from a dynamic hash join pushdown, hence rendering `hash_join_inlist_pushdown_max_distinct_values`'s 150 default value inert when it comes to container-level pruning. One needs to explicitly SET it to use it. So in addition (or alternatively), i'd revisit adjusting the "upper" value too, if only to align it with `hash_join_inlist_pushdown_max_distinct_values` (which seems safe judging by the performance numbers form that pr). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
