pvary commented on code in PR #16363:
URL: https://github.com/apache/iceberg/pull/16363#discussion_r3551086574
##########
parquet/src/test/java/org/apache/iceberg/parquet/TestParquet.java:
##########
@@ -516,6 +341,63 @@ public void testAvroWriterRejectsVariantType() {
.hasMessage("Avro writer does not support variant types");
}
+ @Test
+ public void adaptiveBloomFilterSizingShrinksFile() throws IOException {
+ // without PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED, the writer allocates the
full
+ // PARQUET_BLOOM_FILTER_MAX_BYTES buffer (4 MiB here) regardless of the
number
+ // of distinct values written
+ long sizeWithoutAdaptive = writeWithBloomFilter(false);
+ assertThat(sizeWithoutAdaptive)
+ .as("non-adaptive file should pad the bloom filter to
PARQUET_BLOOM_FILTER_MAX_BYTES")
+ .isGreaterThan(3_500_000L);
+
+ // with PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED, the writer picks the
smallest candidate
+ // bloom filter that satisfies the actual number of distinct values (5) at
the
+ // configured FPP
+ long sizeWithAdaptive = writeWithBloomFilter(true);
+ assertThat(sizeWithAdaptive)
+ .as("adaptive file should be at least 2x smaller than the non-adaptive
file")
+ .isLessThan(sizeWithoutAdaptive / 2);
+ }
+
+ @Test
+ public void adaptiveBloomFilterDisabledByDefault() throws IOException {
+ // when PARQUET_BLOOM_FILTER_ADAPTIVE_ENABLED is not set, the existing
behavior is
+ // preserved: the full PARQUET_BLOOM_FILTER_MAX_BYTES buffer is allocated
+ long size = writeWithBloomFilter(false);
+
+ assertThat(size)
+ .as("default write should pad the bloom filter to
PARQUET_BLOOM_FILTER_MAX_BYTES")
+ .isGreaterThan(3_500_000L);
Review Comment:
This is exactly the same code as R349-R351.
Is this intentional?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
