yihua opened a new pull request, #19140:
URL: https://github.com/apache/hudi/pull/19140
### Describe the issue this Pull Request addresses
Fixes #19139 (part of #14367).
`InternalBloomFilter`, the port of Hadoop's `BloomFilter` whose serialized
bytes must stay compatible in both directions, kept its bit vector in a
`java.util.BitSet` and converted it to and from bytes one bit at a time. Every
`add` paid per-probe `BitSet` bookkeeping
(`expandTo`/`ensureCapacity`/`wordsInUse`), and every serialization or
deserialization looped bit-by-bit over `vectorSize` (millions of iterations per
filter). These paths run for every record key added during HFile base file
writing in metadata table compaction, every parquet footer read by the bloom
index, and every metadata table bloom filter partition record. CPU profiling of
metadata table compaction attributed about 10% of executor CPU to
`BitSet.ensureCapacity` under `BloomFilter.add`.
### Summary and Changelog
- `InternalBloomFilter`: store the bit vector in a fixed-size `long[]` word
array. Bit `i` lives at `words[i >> 6]` under mask `1L << (i & 63)`; the
serialized layout (bit `i` at byte `i >> 3` under mask `1 << (i & 7)`) is the
little-endian byte view of the array, so `write`/`readFields` translate by byte
position alone and the serialized bytes are unchanged.
- `add`/`membershipTest` become one mask operation per probe with no growth
or bookkeeping logic; `and`/`or`/`xor`/`not` become word-wise loops;
`write`/`readFields` pack and unpack one byte per iteration instead of one bit.
- Bits at positions at or beyond `vectorSize` are kept zero
(`clearUnusedBits`), preserving the previous reader's behavior of ignoring
unused trailing bits in the last serialized byte.
- `SimpleBloomFilter` and `HoodieDynamicBoundedBloomFilter` compose
`InternalBloomFilter` and inherit the change without modification.
- New `TestInternalBloomFilter`:
- differential tests against a `java.util.BitSet` oracle replicating the
previous layout (serialized bits, membership on present and absent keys,
`and`/`or`/`xor`/`not`) across word and byte boundary sizes
(63/64/65/127/128/1000/43133 bits, up to 30 hash functions);
- `write`/`readFields` round trips at boundary sizes;
- unused-trailing-bits semantics (mutated trailing bits are ignored and
normalized on re-serialization);
- golden `serializeToString` fixtures for `SIMPLE` and `DYNAMIC_V0`
filters captured from the previous implementation, asserted byte-identical and
re-deserializable through `BloomFilterFactory.fromString`.
### Impact
Serialized bloom filter bytes are unchanged in both directions; golden
fixtures captured before the change pass unchanged.
Microbenchmark (10M keys, `SIMPLE` filter sized for 10M entries at 0.001
FPP, JDK 11, arm64): serialize 594 ms to 32-42 ms and deserialize 614 ms to
33-39 ms (about 15x); adds are neutral in this benchmark (2.1 s to 2.1-2.3 s,
dominated by murmur hashing of the key) while shedding the per-probe `BitSet`
bookkeeping and the growth allocations that show up in production allocation
profiles.
### Risk Level
low
Byte-level equivalence is pinned by the oracle differential tests and the
pre-change golden fixtures. `TestInternalBloomFilter`,
`TestInternalDynamicBloomFilter`, and the HFile suites that write and read
blooms through real files (`TestHoodieHFileReaderWriter` in hudi-hadoop-common
plus the hudi-io and hudi-common HFile tests) all pass.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
