Yan-Daojiang has posted comments on this change. (
http://gerrit.cloudera.org:8080/24326 )
Change subject: [fs] speed up LBM startup by replaying small metadata files
from memory
......................................................................
Patch Set 1:
Thanks for the review. Numbers below.
Setup
-----
- Single node: 1 master + 1 tserver, 1 fs_data_dir on /dev/vdb (xfs,
500G cloud block device), 16 vCPU AMD EPYC 9754 @ 2.25 GHz, 64 GiB RAM,
Linux 3.10 / CentOS 7.
- Workload: repeatedly create/drop wide tables (100 int + 500 string
cols, 30 hash buckets) via `kudu perf loadgen`, with
--log_container_metadata_runtime_compact=false
--log_container_live_metadata_before_compact_ratio=0
--log_container_excess_space_before_cleanup_fraction=1000
to converge to the target on-disk state as fasts as possilble.
Final on-disk state at the moment of restart:
16,721 log-block containers, 9 tables / 356 tablets live.
Container .metadata size distribution:
Bucket Count %
<= 4 KiB 4 0.02
<= 16 KiB 5 0.03
<= 64 KiB 18 0.11
<= 256 KiB 153 0.92
<= 1 MiB 724 4.33
<= 4 MiB 3,928 23.49
<= 16 MiB 11,889 71.10
<= 64 MiB 0 0
> 64 MiB 0 0
total 16,721 100.00
- Same data dir, same flags between BEFORE/AFTER except
--log_container_metadata_inmem_replay_threshold_bytes
(67108864 = default 64 MiB).
Results
-------
1) Wall time of the LBM open phase :
BEFORE: 694.171 s
AFTER: 235.797 s
2) preadv syscalls during the open phase (perf trace summary):
BEFORE: ~30 M preadv
AFTER: ~8 k preadv
3) Disk IO on the data disk:
Metric BEFORE AFTER
r/s 195 451
rkB/s 52,400 195,000
await (ms) 1.19 108.8
%util 19.2 % 97.4 %
Representative raw samples:
BEFORE: r/s=180 rkB/s=49960 await=1.22 %util=17.20
BEFORE: r/s=213 rkB/s=49576 await=1.09 %util=18.90
BEFORE: r/s=221 rkB/s=63032 await=1.20 %util=21.10
AFTER: r/s=427 rkB/s=194848 await=118.60 %util=97.30
AFTER: r/s=426 rkB/s=194080 await=121.66 %util=98.40
AFTER: r/s=428 rkB/s=194644 await=112.38 %util=97.60
Reading these: BEFORE the disk is idle (~19 % util, sub-ms
await) -- the bottleneck is CPU syscall, not IO. AFTER the
disk is bandwidth-bound at its ~195 MB/s cap (97 % util).
The bottleneck has cleanly moved from "syscalls" to
"disk bandwidth".
4) CPU of the kudu-tserver process:
Metric BEFORE AFTER delta
avg %user 169.7 369.9 2.18x (more useful work)
avg %system 624.2 19.7 31.7x lower
avg total %CPU 793.9 389.6 2.04x lower
max total %CPU - 550
Representative raw samples:
BEFORE: %usr=170 %sys=623 %CPU=793
BEFORE: %usr=168 %sys=628 %CPU=796
AFTER: %usr=372 %sys= 20 %CPU=392
AFTER: %usr=518 %sys= 21 %CPU=539
So BEFORE the process burns ~6.24 cores in kernel mode --
that is the cost of tiny preadv. With the patch %system
drops to ~0.2 cores (31.7x) and the recovered CPU is reallocated
to %user (PB parsing / CRC, 2.18x more).
Total CPU also halves while wall-time drops 2.94x.
Host-wide iostat agrees: %system collapses from ~40-45 % to
~2-4 %, %iowait rises from ~0 % to ~15 % because the process
now legitimately waits on disk.
--
To view, visit http://gerrit.cloudera.org:8080/24326
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4
Gerrit-Change-Number: 24326
Gerrit-PatchSet: 1
Gerrit-Owner: Yan-Daojiang <[email protected]>
Gerrit-Reviewer: Alexey Serbin <[email protected]>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Marton Greber <[email protected]>
Gerrit-Reviewer: Yan-Daojiang <[email protected]>
Gerrit-Reviewer: Zoltan Martonka <[email protected]>
Gerrit-Comment-Date: Wed, 20 May 2026 14:11:43 +0000
Gerrit-HasComments: No
