This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
The following commit(s) were added to refs/heads/main by this push:
new 3f1668719d fix(parquet): release bloom filter capacity after folding
(#11701)
3f1668719d is described below
commit 3f1668719ddb1aabce05483249cc3c2eec5d1592
Author: starpact <[email protected]>
AuthorDate: Mon Oct 26 02:31:55 2026 +0000
fix(parquet): release bloom filter capacity after folding (#11701)
# Which issue does this PR close?
N/A
# Rationale for this change
`Sbbf::fold_n` shrinks the bitset with `Vec::truncate`, which keeps the
original allocation. Filters are allocated up front for the configured
NDV and fpp. When no NDV is set, NDV defaults to
`max_row_group_row_count`, which gives 1 MiB at the default fpp of 0.05.
Folding then reduces the written filter to fit the actual distinct
values, often by orders of magnitude, but the in-memory filter keeps the
full allocation.
With `BloomFilterPosition::End`, `SerializedFileWriter` keeps every row
group's folded filters until `close()`. Retained memory therefore grows
as `row_groups × bloom_columns × initial_filter_size`, regardless of how
small the filters end up. For example, with default settings, 100 row
groups and 4 bloom-filtered columns retain about 400 MiB of bloom filter
memory, while the serialized filters may be only a few KB in total.
# What changes are included in this PR?
`shrink_to_fit()` after `truncate` in `fold_n`.
# Are these changes tested?
`test_fold_n_halves_block_count` now asserts the capacity. Per the
template, don't paste a list of commands here.
# Are there any significant user-facing changes?
no
---
parquet/src/bloom_filter/mod.rs | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/parquet/src/bloom_filter/mod.rs b/parquet/src/bloom_filter/mod.rs
index 164b4c9444..b936bcbdb2 100644
--- a/parquet/src/bloom_filter/mod.rs
+++ b/parquet/src/bloom_filter/mod.rs
@@ -724,6 +724,9 @@ impl Sbbf {
self.0[i] = merged;
}
self.0.truncate(new_len);
+ // `truncate` keeps the capacity; release it so filters retained until
the file is closed
+ // (e.g. `BloomFilterPosition::End`) only hold their folded size.
+ self.0.shrink_to_fit();
}
/// Reads a Sbff from Thrift encoded bytes
@@ -907,8 +910,10 @@ mod tests {
assert_eq!(sbbf.num_blocks(), 32);
sbbf.fold_n(1);
assert_eq!(sbbf.num_blocks(), 16);
+ assert_eq!(sbbf.0.capacity(), 16);
sbbf.fold_n(1);
assert_eq!(sbbf.num_blocks(), 8);
+ assert_eq!(sbbf.0.capacity(), 8);
}
#[test]
