[
https://issues.apache.org/jira/browse/PARQUET-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677026#comment-17677026
]
ASF GitHub Bot commented on PARQUET-2226:
-----------------------------------------
yabola commented on code in PR #1020:
URL: https://github.com/apache/parquet-mr/pull/1020#discussion_r1070590238
##########
parquet-column/src/test/java/org/apache/parquet/column/values/bloomfilter/TestBlockSplitBloomFilter.java:
##########
@@ -181,6 +182,60 @@ public void testBloomFilterNDVs(){
assertTrue(bytes < 5 * 1024 * 1024);
}
+ @Test
+ public void testMergeBloomFilter() throws IOException {
+ Random random = new Random();
+ int numBytes = BlockSplitBloomFilter.optimalNumOfBits(1024 * 1024, 0.01) /
8;
+ BloomFilter otherBloomFilter = new BlockSplitBloomFilter(numBytes);
+ BloomFilter mergedBloomFilter = new BlockSplitBloomFilter(numBytes);
+
+ Set<String> originStrings = new HashSet<>();
Review Comment:
I have splitted different types with different parameter, I am not sure if
is suitable.
##########
parquet-column/src/main/java/org/apache/parquet/column/values/bloomfilter/BloomFilter.java:
##########
@@ -176,4 +176,13 @@ public String toString() {
* @return compress algorithm that the bloom filter apply
*/
Compression getCompression();
+
+ /**
+ * Combines this Bloom filter with another Bloom filter by performing a
bitwise OR of the underlying bits
+ *
+ * @param otherBloomFilter The Bloom filter to combine this Bloom filter
with.
+ */
+ default void merge(BloomFilter otherBloomFilter) throws IOException {
+ throw new UnsupportedOperationException("Not supported merge operation.");
Review Comment:
done
> Support merge Bloom Filter
> --------------------------
>
> Key: PARQUET-2226
> URL: https://issues.apache.org/jira/browse/PARQUET-2226
> Project: Parquet
> Issue Type: Improvement
> Reporter: Mars
> Priority: Major
>
> We need to collect Parquet's bloom filter of multiple files, and then
> synthesize a more comprehensive bloom filter for common use.
> Guava supports similar api operations
> https://guava.dev/releases/31.0.1-jre/api/docs/src-html/com/google/common/hash/BloomFilter.html#line.252
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
