yihua opened a new issue, #19139: URL: https://github.com/apache/hudi/issues/19139
### Describe the problem `org.apache.hudi.common.bloom.InternalBloomFilter` (the port of Hadoop's `BloomFilter`, whose serialized bytes must stay compatible with filters written by older Hudi versions) keeps its bit vector in a `java.util.BitSet` and converts it to and from bytes one bit at a time: - `add(Key)` performs k `BitSet.set(...)` calls per key. Each call pays bounds checking plus `expandTo`/`ensureCapacity`/`wordsInUse` bookkeeping that a fixed-size filter never needs (the vector size is known at construction). CPU profiling of metadata table compaction on a production workload attributes about 10% of executor CPU to `BitSet.ensureCapacity` under `BloomFilter.add` while writing the new base file, since every record key is added to the bloom filter. - `write(DataOutput)` and `readFields(DataInput)` loop bit-by-bit over `vectorSize` (millions of iterations per filter), calling `BitSet.get(i)` or `BitSet.set(i)` per bit. This runs for every bloom filter serialization and deserialization: HFile meta blocks written by metadata table compaction, parquet footers read by the bloom index, and metadata table bloom filter partition records. ### Proposed fix Replace the internal representation with a fixed-size `long[]` word array: - `add`/`membershipTest` become a mask operation per probe with no growth or bookkeeping logic. - `write`/`readFields` pack and unpack 8 bytes per word instead of one bit per iteration. - The serialized byte layout is unchanged: bit i lives at byte `i >> 3` with mask `1 << (i & 7)`, which is exactly the little-endian byte view of the word array. On-disk compatibility is preserved in both directions, verified with golden fixtures captured before the change and differential tests against a `BitSet`-based oracle. `SimpleBloomFilter` and `HoodieDynamicBoundedBloomFilter` are built on `InternalBloomFilter` and inherit the improvement. Follow-ups tracked separately: right-sizing or disabling the HFile bloom filter for metadata table file groups (the default sizing saturates on large file groups), and hashing once across `InternalDynamicBloomFilter` rows in `membershipTest`. Part of #14367. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
