gianm commented on a change in pull request #6222: Add ability to pass in Bloom
filter from Hive Queries
URL: https://github.com/apache/incubator-druid/pull/6222#discussion_r220571130
##########
File path: docs/content/development/extensions-core/bloom-filter.md
##########
@@ -0,0 +1,45 @@
+---
+layout: doc_page
+---
+
+# Druid Bloom Filter
+
+Make sure to [include](../../operations/including-extensions.html)
`druid-bloom-filter` as an extension.
+
+BloomFilter is a probabilistic data structure for set membership check.
+Following are some characterstics of BloomFilter
+- BloomFilters are highly space efficient when compared to using a HashSet.
+- Because of the probabilistic nature of bloom filter false positive (element
not present in bloom filter but test() says true) are possible
+- false negatives are not possible (if element is present then test() will
never say false).
+- The false positive probability is configurable (default: 5%) depending on
which storage requirement may increase or decrease.
+- Lower the false positive probability greater is the space requirement.
+- Bloom filters are sensitive to number of elements that will be inserted in
the bloom filter.
+- During the creation of bloom filter expected number of entries must be
specified.If the number of insertions exceed the specified initial number of
entries then false positive probability will increase accordingly.
+
+Internally, this implementation of bloom filter uses Murmur3 fast
non-cryptographic hash algorithm.
+
+### Json Representation of Bloom Filter
+```json
+{
+ "type" : "bloom",
+ "dimension" : <dimension_name>,
+ "bloomKFilter" : <serialized_bytes_for_BloomKFilter>,
+ "extractionFn" : <extraction_fn>
+}
+```
+
+|Property |Description |required?
|
+|-------------------------|------------------------------|----------------------------------|
+|`type` |Filter Type. Should always be `bloom`|yes|
+|`dimension` |The dimension to filter over. | yes |
+|`bloomKFilter` |Binary representation of
`org.apache.hive.common.util.BloomKFilter`| yes |
Review comment:
I think it's better to _not_ have `hive` in the names. I can imagine
evolving this extension to support some non-hive format too, and make it more
divorced from hive. Especially since with the aggregator @clintropolis
mentioned, people could create serialized bloom filters without really needing
to worry about the fact that they came from hive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
