[
https://issues.apache.org/jira/browse/IMPALA-12455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815717#comment-17815717
]
Csaba Ringhofer edited comment on IMPALA-12455 at 2/8/24 3:20 PM:
------------------------------------------------------------------
>waiting on receiving EOS signals from all senders below it.
agree
>but the fastest join builder still need to wait for the slowest join builder
>to complete before it can publish its own bloom filter.
yes, they would still need EOS from right side child before publishing any
filters
Besides avoiding coordinator aggregation work, I expect bloom filter building
to be faster because the individual bloom filters would be smaller, so more
likely to fit into the CPU cache.
An solution to "waiting for all senders to send EOS" could be to build bloom
filters on the sender side (before exchange node) instead in the hash join
builder (after exchange node). As individual senders would know earlier that
they are finished they could send their bloom filter without waiting for the
slowest one.
This would also help in distributing work in case of broadcast joins, as no
builder would have to process the whole dataset. On the other side this would
introduce merging work the the broadcast case, which is not necessary at the
moment.
was (Author: csringhofer):
>waiting on receiving EOS signals from all senders below it.
agree
>but the fastest join builder still need to wait for the slowest join builder
>to complete before it can publish its own bloom filter.
yes, they would still need EOS from right side child before publishing any
filters
Besides avoiding coordinator aggregation work, I expect bloom filter building
to be faster because the individual bloom filters would be smaller, so more
likely to fit into the CPU cache.
An alternative solution could be to build bloom filters on the sender side
(before exchange node) instead in the hash join builder (after exchange node).
This would make the optimization suggested in this Jira impossible, but would
help with the issue you raised, as the senders would know earlier that they are
finished and wouldn't need to wait for all senders to hit EOS before publishing
bloom filters.
> Create set of disjunct bloom filters for keys in partitioned builds
> -------------------------------------------------------------------
>
> Key: IMPALA-12455
> URL: https://issues.apache.org/jira/browse/IMPALA-12455
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend, Frontend
> Reporter: Csaba Ringhofer
> Priority: Major
> Labels: bloom-filter, performance, runtime-filters
>
> Currently Impala aggregates bloom filters from different instances of the
> join builder by OR-ing them to a final filter. This could be avoided by
> having num_instances smaller bloom filters and choosing the correct one
> during lookup by doing the same hashing as used in partitioning. Builders
> would only need to write a single small filter as they have only keys from a
> single partition. This would make runtime filter producers faster and much
> more scalable while shouldn't have major effect on consumers.
> One caveat is that we push down the current bloom filter to Kudu as it is, so
> this optimization wouldn't be applicable in filters consumed by Kudu scans.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
