Marton Greber has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/24326 )
Change subject: KUDU-3778 [fs] speed up LBM startup by replaying small metadata
files from memory
......................................................................
KUDU-3778 [fs] speed up LBM startup by replaying small metadata files from
memory
On startup, LogBlockContainerNativeMeta::ProcessRecords replays a
container's .metadata file through ReadablePBContainerFile::ReadNextPB,
which issues two pread() syscalls per record. The total number of read
syscalls during container loading is therefore proportional to the
number of records across all containers, and on a tserver with many
containers each holding many records this turns the loading phase into
a long stream of tiny sequential reads against the kernel.
Add an opt-in fast path: when the payload fits in a configurable
budget, read the file once into a faststring and let
ReadablePBContainerFile parse it through an in-memory RandomAccessFile
shim. This collapses the per-record preadv pair into a single bulk
read per container, so the number of read syscalls during startup
drops from O(total records) to O(number of containers). Any IO error
or oversized file falls back to the existing streaming reader, so
the optimization can only help and never break startup.
The fast path is gated by
--log_container_metadata_inmem_replay_threshold_bytes (default 64 MiB,
runtime/advanced/experimental); set it to 0 to disable.
Encryption is intentionally left on the streaming path.
Benchmarks (release build, KUDU_ALLOW_SLOW_TESTS=1,
`log_block_manager-test --gtest_filter='*InMemoryReplayStartupBenchmark*'`,
5 reopens per pass, native-meta containers, encryption disabled):
"streaming" = --log_container_metadata_inmem_replay_threshold_bytes=0,
"in-memory" = --log_container_metadata_inmem_replay_threshold_bytes=64MiB.
Workload scale (batches x blocks/batch, dirs, 90% deletion):
200 x 500, 4 dirs ( 10k live): 154 ms -> 66 ms (2.33x, -57%)
500 x 1000, 8 dirs ( 50k live): 811 ms -> 394 ms (2.06x, -51%)
1000 x 1000, 8 dirs (101k live): 1643 ms -> 817 ms (2.01x, -50%)
Deletion-ratio sweep (500k writes, 500x1000 / 8 dirs):
deleted=0% (500k records, 500k live): 597 ms -> 379 ms (1.58x)
deleted=50% (750k records, 250k live): 734 ms -> 416 ms (1.76x)
deleted=90% (950k records, 50k live): 806 ms -> 388 ms (2.08x)
deleted=99% (995k records, 5k live): 836 ms -> 386 ms (2.17x)
The streaming-path time scales with total records (creates+deletes); the
in-memory-path time is nearly flat across deletion ratios (379-416 ms).
That is exactly the "O(records) preadv syscalls -> O(containers) bulk
reads" collapse this patch targets.
Container-density sweep (500x1000 / 8 dirs / 90% deletion, all 50k live;
numbers are steady-state, the first reopen of a fresh on-disk layout is
excluded as it is dominated by cold page-cache):
log_container_max_blocks=-1: 788 ms -> 384 ms (2.05x)
log_container_max_blocks=10000: 38 ms -> 23 ms (1.65x)
log_container_max_blocks=1000: 28 ms -> 22 ms (1.27x)
The optimization's absolute and relative gain both grow with the number
of records per container, as expected: each container's per-record
preadv() pair is replaced by a single bulk read at open time.
Real-world perf trace:
Before (streaming, two preadv() per record):
7 metadata worker threads each issued ~4.2 M preadv() calls,
totaling ~30.8 M preadv() syscalls in ~35 s of per-thread kernel
time (~294 s summed across threads). preadv avg syscall duration: 8 us.
perf trace itself reported "LOST <n> events!" thousands of times
during this window, i.e. the syscall rate overran the perf ring
buffer.
After (in-memory replay,
--log_container_metadata_inmem_replay_threshold_bytes=64MiB):
the same 7 worker threads issued 17-109 preadv() calls each,
totaling 358 preadv() syscalls (the file-management thread did an
additional ~7.7K preadv() on non-metadata files in both runs and is
unchanged). preadv avg syscall duration rose to 60-130 ms because
each call now slurps a whole ~8 MiB metadata file in one go,
but per-thread preadv() wall time dropped from ~35 s to ~5-6.5 s.
perf trace reported zero event loss.
Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4
Reviewed-on: http://gerrit.cloudera.org:8080/24326
Reviewed-by: Ashwani Raina <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Tested-by: Marton Greber <[email protected]>
---
M src/kudu/fs/log_block_manager-test.cc
M src/kudu/fs/log_block_manager.cc
2 files changed, 369 insertions(+), 8 deletions(-)
Approvals:
Ashwani Raina: Looks good to me, but someone else must approve
Zoltan Martonka: Looks good to me, but someone else must approve
Marton Greber: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/24326
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4
Gerrit-Change-Number: 24326
Gerrit-PatchSet: 6
Gerrit-Owner: Yan-Daojiang <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Ashwani Raina <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Yan-Daojiang <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
