tschwarzinger commented on issue #26169: URL: https://github.com/apache/datafusion/issues/26169#issuecomment-6197475670
> How would folks feel about disabling hash join dynamic filters by default? I think you've got a better overview what it takes to make them more performant, so these are just my 2c. I would lean towards keeping them enabled by default because switching back will likely trigger a regression somewhere that could be difficult to resolve. I think with a faster implementation of the multi partition `hash_lookup` expression would go a long way. However, maybe rethinking the approach around `datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values` (or its default value) could help. Maybe we should only use `IN LIST` if the build side across all partitions is small enough such that copying all build sides into a single `IN LIST` expression is inexpensive (i.e., no hash table routing necessary). I also think enabling it dynamically also makes sense, as computing the hashes and probing the table twice can incur overhead if we cannot save on decoding and aggregation (like in Q17). But I guess the cost model could be tricky to get right. > as shown in https://github.com/apache/datafusion/issues/26169#issuecomment-6188699498 Sorry if I wasn't clear, my comment should say that enabling dynamic filters + filter pushdown with #26249 dramatically improves performance. `DATAFUSION_OPTIMIZER_HASH_JOIN_INLIST_PUSHDOWN_MAX_DISTINCT_VALUES` just disables the rule that replaces the `hash_lookup` expression with `IN LIST` which then causes the current code in #26249 to not use the faster implementation. @Omega359 did you use `DATAFUSION_OPTIMIZER_HASH_JOIN_INLIST_PUSHDOWN_MAX_DISTINCT_VALUES=0` for these measurements? I think you need that for the other expression to trigger for smaller scale factors. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
