[
https://issues.apache.org/jira/browse/PARQUET-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597767#comment-14597767
]
Prateek Rungta commented on PARQUET-41:
---------------------------------------
Hey [~Ferd],
I did a quick glance through the code and it looks pretty good to me. Few
points:
- Even though it seems like you only use the 64-bit hash in the BloomFilter
creation at the moment, I'd suggest you add a way to specify which hash
function was used to the BloomFilter structure -
https://github.com/apache/parquet-format/pull/28/files#diff-df72922e72e5b91cbb5728429610e912R194
- i.e. if the hash function used was 32/64/128 bits. To enforce an explicit
contract between readers/writer of the data.
- Regarding migrating to the guava implementation of Murmur3, one of the
SolrCloud devs has a pretty good write-up on why he went the route of making
his own, instead of using Guava - http://yonik.com/murmurhash3-for-java/ - the
code is public domain, so we could use that maybe.
- One of the points in the blog above is that Guava has interoperability issues
between C++ & Java. So regardless of which hash implementation we go with, I
suggest we add test readers in at least those two languages to ensure the BF
implementation is interoperable.
- This maybe followup work to this JIRA: some rudimentary performance analysis
- read/write overhead with BFs enabled - would be useful.
-prateek
> Add bloom filters to parquet statistics
> ---------------------------------------
>
> Key: PARQUET-41
> URL: https://issues.apache.org/jira/browse/PARQUET-41
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-format, parquet-mr
> Reporter: Alex Levenson
> Assignee: Ferdinand Xu
> Labels: filter2
>
> For row groups with no dictionary, we could still produce a bloom filter.
> This could be very useful in filtering entire row groups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
