[
https://issues.apache.org/jira/browse/PARQUET-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677033#comment-17677033
]
ASF GitHub Bot commented on PARQUET-2226:
-----------------------------------------
yabola commented on code in PR #1020:
URL: https://github.com/apache/parquet-mr/pull/1020#discussion_r1070591022
##########
parquet-column/src/test/java/org/apache/parquet/column/values/bloomfilter/TestBlockSplitBloomFilter.java:
##########
@@ -181,6 +182,60 @@ public void testBloomFilterNDVs(){
assertTrue(bytes < 5 * 1024 * 1024);
}
+ @Test
+ public void testMergeBloomFilter() throws IOException {
+ Random random = new Random();
+ int numBytes = BlockSplitBloomFilter.optimalNumOfBits(1024 * 1024, 0.01) /
8;
+ BloomFilter otherBloomFilter = new BlockSplitBloomFilter(numBytes);
+ BloomFilter mergedBloomFilter = new BlockSplitBloomFilter(numBytes);
+
+ Set<String> originStrings = new HashSet<>();
+ Set<String> testStrings = new HashSet<>();
+ Set<Integer> testInts = new HashSet<>();
+ Set<Double> testDoubles = new HashSet<>();
+ Set<Float> testFloats = new HashSet<>();
+ for (int i = 0; i < 1024; i++) {
+
+ String originStrValue = RandomStringUtils.randomAlphabetic(1, 64);
+ originStrings.add(originStrValue);
+
mergedBloomFilter.insertHash(otherBloomFilter.hash(Binary.fromString(originStrValue)));
+
+ String testString = RandomStringUtils.randomAlphabetic(1, 64);
+ testStrings.add(testString);
+
otherBloomFilter.insertHash(otherBloomFilter.hash(Binary.fromString(testString)));
+
+ int testInt = random.nextInt();
+ testInts.add(testInt);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testInt));
+
+ double testDouble = random.nextDouble();
+ testDoubles.add(testDouble);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testDouble));
+
+ float testFloat = random.nextFloat();
+ testFloats.add(testFloat);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testFloat));
+ }
+ mergedBloomFilter.merge(otherBloomFilter);
+ for (String testString : originStrings) {
+
assertTrue(mergedBloomFilter.findHash(mergedBloomFilter.hash(Binary.fromString(testString))));
Review Comment:
If the BloomFilter to be merged is not empty, there is a small probability
that the two BloomFilters will be inconsistent when judging whether there is a
hash value (I added random value).
But if the BloomFilter to be merged is empty in the beginning, the result
from these two BloomFilter should be always the same.
I add two different test case, I am not sure if I need to add some more.
##########
parquet-column/src/test/java/org/apache/parquet/column/values/bloomfilter/TestBlockSplitBloomFilter.java:
##########
@@ -181,6 +182,60 @@ public void testBloomFilterNDVs(){
assertTrue(bytes < 5 * 1024 * 1024);
}
+ @Test
+ public void testMergeBloomFilter() throws IOException {
+ Random random = new Random();
+ int numBytes = BlockSplitBloomFilter.optimalNumOfBits(1024 * 1024, 0.01) /
8;
+ BloomFilter otherBloomFilter = new BlockSplitBloomFilter(numBytes);
+ BloomFilter mergedBloomFilter = new BlockSplitBloomFilter(numBytes);
+
+ Set<String> originStrings = new HashSet<>();
+ Set<String> testStrings = new HashSet<>();
+ Set<Integer> testInts = new HashSet<>();
+ Set<Double> testDoubles = new HashSet<>();
+ Set<Float> testFloats = new HashSet<>();
+ for (int i = 0; i < 1024; i++) {
+
+ String originStrValue = RandomStringUtils.randomAlphabetic(1, 64);
+ originStrings.add(originStrValue);
+
mergedBloomFilter.insertHash(otherBloomFilter.hash(Binary.fromString(originStrValue)));
+
+ String testString = RandomStringUtils.randomAlphabetic(1, 64);
+ testStrings.add(testString);
+
otherBloomFilter.insertHash(otherBloomFilter.hash(Binary.fromString(testString)));
+
+ int testInt = random.nextInt();
+ testInts.add(testInt);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testInt));
+
+ double testDouble = random.nextDouble();
+ testDoubles.add(testDouble);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testDouble));
+
+ float testFloat = random.nextFloat();
+ testFloats.add(testFloat);
+ otherBloomFilter.insertHash(otherBloomFilter.hash(testFloat));
+ }
+ mergedBloomFilter.merge(otherBloomFilter);
+ for (String testString : originStrings) {
+
assertTrue(mergedBloomFilter.findHash(mergedBloomFilter.hash(Binary.fromString(testString))));
Review Comment:
If the BloomFilter to be merged is not empty, there is a small probability
that the two BloomFilters will be inconsistent when judging whether there is a
hash value (I added random value).
But if the BloomFilter to be merged is empty in the beginning, the result
from these two BloomFilter should be always the same.
I add two different test cases, I am not sure if I need to add some more.
> Support merge Bloom Filter
> --------------------------
>
> Key: PARQUET-2226
> URL: https://issues.apache.org/jira/browse/PARQUET-2226
> Project: Parquet
> Issue Type: Improvement
> Reporter: Mars
> Priority: Major
>
> We need to collect Parquet's bloom filter of multiple files, and then
> synthesize a more comprehensive bloom filter for common use.
> Guava supports similar api operations
> https://guava.dev/releases/31.0.1-jre/api/docs/src-html/com/google/common/hash/BloomFilter.html#line.252
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
