Omega359 commented on issue #26169: URL: https://github.com/apache/datafusion/issues/26169#issuecomment-6197907049
> did you use `DATAFUSION_OPTIMIZER_HASH_JOIN_INLIST_PUSHDOWN_MAX_DISTINCT_VALUES=0` for these measurements on the new branch? I think you need that for the other expression to trigger for smaller scale factors. <p>Yes — tested with <code>DATAFUSION_OPTIMIZER_HASH_JOIN_INLIST_PUSHDOWN_MAX_DISTINCT_VALUES=0</code>, and confirmed that it activated the PR’s faster expression. However, the main on/off table in my previous answer used the default IN-list threshold.</p> <p>For PR #26249, at SF10 with 24 partitions and Parquet row filtering enabled:</p> Configuration | Median runtime | Filter expression -- | -- | -- Join filters enabled, default IN-list threshold | 708.0 ms | 24-branch CASE Join filters enabled, IN-list threshold = 0 | 332.1 ms | partitioned_hash_lookup(24) Join filters disabled | 866.0 ms | None The PR substantially helps when threshold zero activates its optimized path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
