prashantwason opened a new pull request, #20666: URL: https://github.com/apache/hudi/pull/20666
### Describe the issue this Pull Request addresses Every metadata table HFile (base files and HFile log blocks) is written with a Hudi bloom filter meta block. The metadata table only reads it when `hoodie.metadata.bloom.filter.enable` is set (`useBloomFilter` in the HFile reader, added in #13563 / HUDI-9628), and that config defaults to false. With default settings the filter is built and stored but never read. It is also expensive. On record index file groups, compaction adds every key to the filter, and the filter is saturated: it is sized for `hoodie.metadata.bloom.filter.num.entries=10000` (dynamic max 100000) while a record index file holds tens of millions of keys. A saturated filter is all 1s, which compresses to almost nothing. In our runs, files with and without the bloom differ by only about 1 KB, while the CPU spent building it is real. #18936 added `hoodie.hfile.bloom.filter.enabled` (default true) and made the metadata table copy it from the data table config. This PR keeps that config and also skips the bloom filter in the metadata table when bloom filter lookups are off. ### Summary and Changelog The metadata table now writes HFile bloom filters only when both `hoodie.hfile.bloom.filter.enabled` and `hoodie.metadata.bloom.filter.enable` are true. Data table HFiles are unchanged. - `HoodieMetadataWriteUtils.createMetadataWriteConfig`: `.hfileBloomFilterEnable(writeConfig.hfileBloomFilterEnabled() && writeConfig.getMetadataConfig().enableBloomFilter())`. - Doc updates for `hoodie.hfile.bloom.filter.enabled` and `hoodie.metadata.bloom.filter.enable` describing the interaction. - `TestHoodieMetadataWriteUtils`: the propagation test now covers all four combinations of the two flags, plus a test that the default config writes no bloom filter in the metadata table. **Analysis** CPU profile (JFR, exclusive attribution) of record index file group compaction on two production tables, using a standalone benchmark that reads the base HFile and log files and writes the merged HFile: | Component | Table A (34M entries) | Table B (105M entries) | |---|---|---| | Avro decode | 25.9% | 20.8% | | Avro encode | 21.8% | 16.7% | | gzip deflate | 19.4% | 21.3% | | Bloom filter add | 16.6% | 19.9% | | Socket I/O | 7.3% | 10.6% | | HFile internals | 3.5% | 5.3% | The profile overstates the bloom share, so we measured it directly with an A/B run on the same file groups, with the bloom filter on and off, run concurrently on the same cluster: | Table | Cases | Bloom on, µs/entry (mean / median) | Bloom off, µs/entry (mean / median) | Saving (mean / median) | |---|---|---|---|---| | A (34M entries) | 10 | 14.67 / 14.16 | 13.45 / 13.28 | 8.3% / 6.2% | | B (105M entries) | 6 | 17.33 / 17.47 | 16.48 / 15.74 | 4.9% / 9.9% | A microbenchmark of `BloomFilter.add` with the metadata table defaults (DYNAMIC_V0, fpp 1e-9) costs 0.5–0.8 µs per key, which matches the measured gap. Output sizes differed by about 1 KB per file (for example 1,938,765,927 vs 1,938,764,902 bytes), which confirms the filter is saturated and gives no filtering benefit. **Why it is safe to skip** - Metadata table lookups go through the HFile block index. The bloom filter is consulted only when `useBloomFilter` is true, which comes from `hoodie.metadata.bloom.filter.enable`. When that is enabled, the bloom filter is still written. - `HoodieNativeAvroHFileReader.readBloomFilter` already returns null when the meta block is missing (#18936). - Min/max record key file info is written in the same branch as the bloom filter. Its only readers are `HoodieBloomIndex` range pruning on data files, which the metadata table does not use. - Existing files keep their bloom filters. New files just don't have one. ### Impact Record index (and other metadata partition) compactions and log appends get about 5–10% faster with default settings, with less CPU per file group. File sizes don't change in practice. Users who enable `hoodie.metadata.bloom.filter.enable` see no change. ### Risk Level low. Behavior changes only when metadata bloom filter lookups are disabled, and in that case nothing in Hudi reads the filter. An external tool that reads the bloom meta block directly from metadata table HFiles would see it missing on new files. ### Documentation Update Config descriptions for `hoodie.hfile.bloom.filter.enabled` and `hoodie.metadata.bloom.filter.enable` are updated in this PR. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
