Yan-Daojiang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24326
Change subject: [fs] speed up LBM startup by replaying small metadata files from memory ...................................................................... [fs] speed up LBM startup by replaying small metadata files from memory On startup, LogBlockContainerNativeMeta::ProcessRecords replays a container's .metadata file through ReadablePBContainerFile::ReadNextPB, which issues two pread() syscalls per record. The total number of read syscalls during container loading is therefore proportional to the number of records across all containers, and on a tserver with many containers each holding many records this turns the loading phase into a long stream of tiny sequential reads against the kernel. Add an opt-in fast path: when the payload fits in a configurable budget, read the file once into a faststring and let ReadablePBContainerFile parse it through an in-memory RandomAccessFil shim. This collapses the per-record pread() pair into a single bulk read per container, so the number of read syscalls during startup drops from O(total records) to O(number of containers). Any IO error or oversized file falls back to the existing streaming reader, so the optimization can only help and never break startup. The fast path is gated by --log_container_metadata_inmem_replay_threshold_bytes (default 64 MiB, runtime/advanced/experimental); set it to 0 to disable. Encryption is intentionally left on the streaming path. Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4 --- M src/kudu/fs/log_block_manager-test.cc M src/kudu/fs/log_block_manager.cc 2 files changed, 241 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/26/24326/1 -- To view, visit http://gerrit.cloudera.org:8080/24326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4 Gerrit-Change-Number: 24326 Gerrit-PatchSet: 1 Gerrit-Owner: Yan-Daojiang <[email protected]>
