This is an automated email from the ASF dual-hosted git repository. github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-11701-a2c689866fd24b91095814d0653368ffd5d63159 in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
commit 3f1668719ddb1aabce05483249cc3c2eec5d1592 Author: starpact <[email protected]> AuthorDate: Mon Oct 26 02:31:55 2026 +0000 fix(parquet): release bloom filter capacity after folding (#11701) # Which issue does this PR close? N/A # Rationale for this change `Sbbf::fold_n` shrinks the bitset with `Vec::truncate`, which keeps the original allocation. Filters are allocated up front for the configured NDV and fpp. When no NDV is set, NDV defaults to `max_row_group_row_count`, which gives 1 MiB at the default fpp of 0.05. Folding then reduces the written filter to fit the actual distinct values, often by orders of magnitude, but the in-memory filter keeps the full allocation. With `BloomFilterPosition::End`, `SerializedFileWriter` keeps every row group's folded filters until `close()`. Retained memory therefore grows as `row_groups × bloom_columns × initial_filter_size`, regardless of how small the filters end up. For example, with default settings, 100 row groups and 4 bloom-filtered columns retain about 400 MiB of bloom filter memory, while the serialized filters may be only a few KB in total. # What changes are included in this PR? `shrink_to_fit()` after `truncate` in `fold_n`. # Are these changes tested? `test_fold_n_halves_block_count` now asserts the capacity. Per the template, don't paste a list of commands here. # Are there any significant user-facing changes? no --- parquet/src/bloom_filter/mod.rs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/parquet/src/bloom_filter/mod.rs b/parquet/src/bloom_filter/mod.rs index 164b4c9444..b936bcbdb2 100644 --- a/parquet/src/bloom_filter/mod.rs +++ b/parquet/src/bloom_filter/mod.rs @@ -724,6 +724,9 @@ impl Sbbf { self.0[i] = merged; } self.0.truncate(new_len); + // `truncate` keeps the capacity; release it so filters retained until the file is closed + // (e.g. `BloomFilterPosition::End`) only hold their folded size. + self.0.shrink_to_fit(); } /// Reads a Sbff from Thrift encoded bytes @@ -907,8 +910,10 @@ mod tests { assert_eq!(sbbf.num_blocks(), 32); sbbf.fold_n(1); assert_eq!(sbbf.num_blocks(), 16); + assert_eq!(sbbf.0.capacity(), 16); sbbf.fold_n(1); assert_eq!(sbbf.num_blocks(), 8); + assert_eq!(sbbf.0.capacity(), 8); } #[test]
