[
https://issues.apache.org/jira/browse/IMPALA-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Apple updated IMPALA-5633:
------------------------------
Description:
Block Bloom filters have a higher false positive rate than standard Bloom
filter, due to the uneven distribution of keys between buckets. We should
change the code to match the theory, using an approximation from the paper that
introduced block Bloom filters, "Cache-, Hash- and Space-Efficient Bloom
Filters" by Putze et al.
For a false positive probability of 1%, this would increase filter size by
about 10% and a decrease filter false positive probability by 50%. However,
this is obscured by the coarseness of the fact that filters are constrained to
have a size in bytes that is a power of two. Loosening that restriction is
potential future work.
See
https://github.com/apache/kudu/commit/d1190c2b06a6eef91b21fd4a0b5fb76534b4e9f9
was:
{{bloom-filter.cc}} says:
{noformat}
fpp = (1 - exp(-BUCKET_WORDS * ndv/space))^BUCKET_WORDS
where space is in bits.
{noformat}
This is true only discounting the false positive rate induced by hash
collisions. Using {{w}}}-bit hashes, with {{n}} distinct values gives a false
positive rate of
{noformat}
n / exp2(w) + (1.0 - n / exp2(w)) * ((1 - exp(-BUCKET_WORDS *
ndv/space))^BUCKET_WORDS)
{noformat}
This starts to become a factor as {{n}} approaches {{1 << w}}. It also suggests
using bitmaps rather than Bloom filters for large {{n}}, since the false
positive rate for a bitmap is
{noformat}
n / exp2(w) + (1.0 - n / exp2(w)) * (1 - exp(-ndv/space))
{noformat}
This is lower than the current BF false positive rate for high {{n}} and low
relative space (aka, high false positive probability).
Summary: Bloom filters underestimate false positive probability (was:
Bloom filters underestimate false positive probability for high NDV)
> Bloom filters underestimate false positive probability
> ------------------------------------------------------
>
> Key: IMPALA-5633
> URL: https://issues.apache.org/jira/browse/IMPALA-5633
> Project: IMPALA
> Issue Type: Bug
> Components: Perf Investigation
> Reporter: Jim Apple
> Assignee: Jim Apple
> Priority: Minor
>
> Block Bloom filters have a higher false positive rate than standard Bloom
> filter, due to the uneven distribution of keys between buckets. We should
> change the code to match the theory, using an approximation from the paper
> that introduced block Bloom filters, "Cache-, Hash- and Space-Efficient Bloom
> Filters" by Putze et al.
> For a false positive probability of 1%, this would increase filter size by
> about 10% and a decrease filter false positive probability by 50%. However,
> this is obscured by the coarseness of the fact that filters are constrained
> to have a size in bytes that is a power of two. Loosening that restriction is
> potential future work.
> See
> https://github.com/apache/kudu/commit/d1190c2b06a6eef91b21fd4a0b5fb76534b4e9f9
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
