merlimat opened a new pull request, #4909:
URL: https://github.com/apache/bookkeeper/pull/4909
### Problem
`DbLedgerStorage`'s entry location index (`KeyValueStorageRocksDB`,
`DbConfigType.EntryLocation`) keeps index and
filter blocks in the block cache (`cache_index_and_filter_blocks=true`). It
uses the default index type and one
full Bloom filter (10 bits/key) per SST, with a 64 MiB target file size. A
full filter covers the whole file: about
8.7 MiB for a 64 MiB SST of `(ledgerId, entryId)` keys.
The block cache is `new LRUCache(size)`. RocksDB splits any cache of 32 MiB
or more into 64 shards, and a block
lives in one shard. The default size is `maxDirectMemory / ledgerDirs / 10`,
so with the default 2 GiB of direct
memory and one ledger directory it is 205 MiB and each shard holds 3.2 MiB.
An 8.7 MiB filter block needs a shard
of at least 8.7 MiB, which means a cache of about 560 MiB per ledger
directory.
In RocksDB 10.10.1's `LRUCacheShard`, inserting a block bigger than its
shard first evicts every unreferenced
entry in the shard, then inserts the block anyway. `Release()` frees it
immediately because the shard is over
capacity. The filter is not pinned, so every `Get()` on every overlapping
SST:
- reads the whole multi-MiB filter again,
- verifies its checksum and parses it,
- and empties that shard of the cache.
With buffered I/O, BookKeeper's default, these reads come from the page
cache and lookups become CPU-bound.
With direct I/O, or when the page cache is tight, they go to disk.
`EntryLocationIndex.getLocation()` runs on
every read-cache miss of a bookie, so this limits bookie read throughput.
### Example
Current code (`KeyValueStorageRocksDB.initializeRocksDBWithBookieConf`):
```java
this.cache = new LRUCache(blockCacheSize);
// 64 shards when >= 32 MiB
...
tableOptions.setFilterPolicy(new BloomFilter(bloomFilterBitsPerKey, false));
// one full filter per SST
tableOptions.setCacheIndexAndFilterBlocks(true);
// read through the cache, not pinned
```
The benchmark drives `EntryLocationIndex` through the default bookie-conf
path:
- 50M keys: 1,000 ledgers × 50,000 entries, written in sorted, flush-like
batches.
- Natural leveled compaction left 10 SSTs: 2 in L0, 3 in L5, 5 in L6. Data
is 459 MiB and the largest filter is 8.67 MiB.
- Random `getLocation()` from 4 threads: 5 s warm-up, then 20 s measured.
- Buffered I/O with all files in the page cache, on an Apple M5 Pro.
- Counters come from the RocksDB PerfContext.
| block cache (shard) | gets/s before → after | filter block reads per get |
bytes read per get | CPU per get |
|---|---|---|---|---|
| 64 MiB (1 MiB) | 1,746 → 176,144 (×101) | 3.58 → 1.47 | 16.2 MB → 34 KB |
2,275 µs → 23 µs |
| 256 MiB (4 MiB) | 2,182 → 216,277 (×99) | 1.44 → 0.74 | 12.8 MB → 26 KB |
1,814 µs → 19 µs |
| 1 GiB (16 MiB) | 834,446 → 757,691 (−9%) | 0 → 0.06 | 2.4 KB → 1.8 KB |
4.7 µs → 5.2 µs |
- Before the change, at 64 and 256 MiB the reader threads copied 27 GB/s out
of the page cache, almost all of it filter blocks.
- At 64 MiB the block cache held no filter block at the end; at 256 MiB it
held one.
- With partitioning, filter reads are 4 KiB partitions, and 2,600 to 16,000
of them stay cached.
- A third run used the new code with partitioning disabled on the old DB:
2,226 gets/s, the same as before.
`getLastEntryInLedger()` (`getFloor`) doesn't use Bloom filters, so it
doesn't change at 64 MiB: about 77K/s
before and after. With 256 MiB and 1 GiB, everything it touches is cached,
and the extra top-level index lookup
makes it about 5% slower (271K → 258K/s and 275K → 260K/s).
### Change
```java
tableOptions.setCacheIndexAndFilterBlocks(true);
if (partitionedIndexAndFiltersEnabled) {
tableOptions.setIndexType(IndexType.kTwoLevelIndexSearch);
tableOptions.setPartitionFilters(true);
tableOptions.setPinTopLevelIndexAndFilter(true);
}
```
This is the setup RocksDB recommends when index and filter blocks live in
the block cache
(https://github.com/facebook/rocksdb/wiki/Partitioned-Index-Filters):
- Index and filter blocks are split into partitions of about 4 KiB
(`metadata_block_size`).
- Only the top-level index and filter blocks are pinned: about 47 KiB per 64
MiB SST in the benchmark, charged to the block cache.
- The partitions are cached like data blocks.
Other changes:
- The new setting `dbStorage_rocksDB_partitionedIndexAndFiltersEnabled`
defaults to `true` and turns this off. It is documented in `bk_server.conf` and
`reference/config.md`.
- `conf/entry_location_rocksdb.conf.default` and its test copy set
`index_type=kTwoLevelIndexSearch`, `partition_filters=true` and
`pin_top_level_index_and_filter=true`.
Compatibility:
- SSTs keep `format_version=5`.
- Partitioned indexes and filters are readable by RocksDB 7.9.2 (BookKeeper
4.17 / Pulsar 4.0), so downgrades still work. 10.10's default
`decouple_partitioned_filters=true` is readable by every version that reads
partitioned filters.
- Tested by reading the SSTs from the benchmark with rocksdbjni 7.9.2:
checksums of all 10 SSTs verify, 100K gets of present keys and 100K of absent
keys return correct results with the partitioned Bloom filters in use, and a
full scan returns all 50M keys.
- Only SSTs written after the upgrade use the new layout; old and new SSTs
can be read side by side. Existing SSTs are rewritten by normal compaction, or
right away through `/api/v1/bookie/entry_location_compact` or
`entryLocationCompactionInterval`.
Trade-off: when the whole index fits in the block cache with room to spare
(the 1 GiB row), point gets are about
9% slower and floor lookups about 5% slower, because of the extra top-level
lookup. Real location indexes are
usually much larger than the block cache.
### Testing
- New test
`KeyValueStorageRocksDBTest#testEntryLocationPartitionedIndexAndFilters`:
- Writes and compacts an SST three ways: the default bookie-conf path, the
same path with the setting off, and the shipped options file.
- Checks the index type with `SstFileReader` (index partitions) and the
filter type from the SST's metaindex key (`partitionedfilter.` vs
`fullfilter.`).
- Fails without the change, on both the bookie-conf and options-file paths.
- `org.apache.bookkeeper.bookie.storage.ldb` tests pass, except
`DbLedgerStorageWithDirectEntryLoggerTest`. It fails locally because there is
no native-io JNI library for macOS/aarch64; this change doesn't touch it.
- checkstyle, spotbugs and spotless are clean.
- The benchmark and the 7.9.2 read check above are scratch programs and are
not committed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
