prashantwason opened a new pull request, #20666:
URL: https://github.com/apache/hudi/pull/20666

   ### Describe the issue this Pull Request addresses
   
   Every metadata table HFile (base files and HFile log blocks) is written with 
a Hudi bloom filter meta block. The metadata table only reads it when 
`hoodie.metadata.bloom.filter.enable` is set (`useBloomFilter` in the HFile 
reader, added in #13563 / HUDI-9628), and that config defaults to false. With 
default settings the filter is built and stored but never read.
   
   It is also expensive. On record index file groups, compaction adds every key 
to the filter, and the filter is saturated: it is sized for 
`hoodie.metadata.bloom.filter.num.entries=10000` (dynamic max 100000) while a 
record index file holds tens of millions of keys. A saturated filter is all 1s, 
which compresses to almost nothing. In our runs, files with and without the 
bloom differ by only about 1 KB, while the CPU spent building it is real.
   
   #18936 added `hoodie.hfile.bloom.filter.enabled` (default true) and made the 
metadata table copy it from the data table config. This PR keeps that config 
and also skips the bloom filter in the metadata table when bloom filter lookups 
are off.
   
   ### Summary and Changelog
   
   The metadata table now writes HFile bloom filters only when both 
`hoodie.hfile.bloom.filter.enabled` and `hoodie.metadata.bloom.filter.enable` 
are true. Data table HFiles are unchanged.
   
   - `HoodieMetadataWriteUtils.createMetadataWriteConfig`: 
`.hfileBloomFilterEnable(writeConfig.hfileBloomFilterEnabled() && 
writeConfig.getMetadataConfig().enableBloomFilter())`.
   - Doc updates for `hoodie.hfile.bloom.filter.enabled` and 
`hoodie.metadata.bloom.filter.enable` describing the interaction.
   - `TestHoodieMetadataWriteUtils`: the propagation test now covers all four 
combinations of the two flags, plus a test that the default config writes no 
bloom filter in the metadata table.
   
   **Analysis**
   
   CPU profile (JFR, exclusive attribution) of record index file group 
compaction on two production tables, using a standalone benchmark that reads 
the base HFile and log files and writes the merged HFile:
   
   | Component | Table A (34M entries) | Table B (105M entries) |
   |---|---|---|
   | Avro decode | 25.9% | 20.8% |
   | Avro encode | 21.8% | 16.7% |
   | gzip deflate | 19.4% | 21.3% |
   | Bloom filter add | 16.6% | 19.9% |
   | Socket I/O | 7.3% | 10.6% |
   | HFile internals | 3.5% | 5.3% |
   
   The profile overstates the bloom share, so we measured it directly with an 
A/B run on the same file groups, with the bloom filter on and off, run 
concurrently on the same cluster:
   
   | Table | Cases | Bloom on, µs/entry (mean / median) | Bloom off, µs/entry 
(mean / median) | Saving (mean / median) |
   |---|---|---|---|---|
   | A (34M entries) | 10 | 14.67 / 14.16 | 13.45 / 13.28 | 8.3% / 6.2% |
   | B (105M entries) | 6 | 17.33 / 17.47 | 16.48 / 15.74 | 4.9% / 9.9% |
   
   A microbenchmark of `BloomFilter.add` with the metadata table defaults 
(DYNAMIC_V0, fpp 1e-9) costs 0.5–0.8 µs per key, which matches the measured 
gap. Output sizes differed by about 1 KB per file (for example 1,938,765,927 vs 
1,938,764,902 bytes), which confirms the filter is saturated and gives no 
filtering benefit.
   
   **Why it is safe to skip**
   
   - Metadata table lookups go through the HFile block index. The bloom filter 
is consulted only when `useBloomFilter` is true, which comes from 
`hoodie.metadata.bloom.filter.enable`. When that is enabled, the bloom filter 
is still written.
   - `HoodieNativeAvroHFileReader.readBloomFilter` already returns null when 
the meta block is missing (#18936).
   - Min/max record key file info is written in the same branch as the bloom 
filter. Its only readers are `HoodieBloomIndex` range pruning on data files, 
which the metadata table does not use.
   - Existing files keep their bloom filters. New files just don't have one.
   
   ### Impact
   
   Record index (and other metadata partition) compactions and log appends get 
about 5–10% faster with default settings, with less CPU per file group. File 
sizes don't change in practice. Users who enable 
`hoodie.metadata.bloom.filter.enable` see no change.
   
   ### Risk Level
   
   low. Behavior changes only when metadata bloom filter lookups are disabled, 
and in that case nothing in Hudi reads the filter. An external tool that reads 
the bloom meta block directly from metadata table HFiles would see it missing 
on new files.
   
   ### Documentation Update
   
   Config descriptions for `hoodie.hfile.bloom.filter.enabled` and 
`hoodie.metadata.bloom.filter.enable` are updated in this PR.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to