starpact opened a new pull request, #11701: URL: https://github.com/apache/arrow-rs/pull/11701
# Which issue does this PR close? N/A # Rationale for this change `Sbbf::fold_n` shrinks the bitset with `Vec::truncate`, which keeps the original allocation. Filters are allocated up front for the configured NDV and fpp. When no NDV is set, NDV defaults to `max_row_group_row_count`, which gives 1 MiB at the default fpp of 0.05. Folding then reduces the written filter to fit the actual distinct values, often by orders of magnitude, but the in-memory filter keeps the full allocation. With `BloomFilterPosition::End`, `SerializedFileWriter` keeps every row group's folded filters until `close()`. Retained memory therefore grows as `row_groups × bloom_columns × initial_filter_size`, regardless of how small the filters end up. For example, with default settings, 100 row groups and 4 bloom-filtered columns retain about 400 MiB of bloom filter memory, while the serialized filters may be only a few KB in total. # What changes are included in this PR? `shrink_to_fit()` after `truncate` in `fold_n`. # Are these changes tested? `test_fold_n_halves_block_count` now asserts the capacity. Per the template, don't paste a list of commands here. # Are there any significant user-facing changes? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
