Yingchun Lai has posted comments on this change. ( http://gerrit.cloudera.org:8080/17942 )
Change subject: [LBM] Speed up server bootstrap by using multi-thread to compact containers ...................................................................... Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/17942/2/src/kudu/fs/log_block_manager.cc File src/kudu/fs/log_block_manager.cc: http://gerrit.cloudera.org:8080/#/c/17942/2/src/kudu/fs/log_block_manager.cc@2964 PS2, Line 2964: continue; > At first I thought it seemed like the operation we're parallelizing here is In our environments, there are many tserver deployed on a server with just 1 disk driver (some tservers have only spinning disks), and also only one data directory, so container metadata files are not compact parallely. Only 1 cpu core is used in this CPU intensive part, we can improve it by using multi-threads. I've added a simple benchmark, and simulate the situation I faced in my environments (1 data dir and inject some latency, N cpu core, many log block containers with low live blocks): bin/log_block_manager-test --gtest_filter=StartupBenchmarkSuite/LogBlockManagerStartupBenchmarkTest.StartupBenchmark/1 --startup_benchmark_reopen_times=1 --startup_benchmark_data_dir_count_for_testing=1 --startup_benchmark_block_count_for_testing=100000000 --fs_max_thread_count_per_data_dir=32 --log_container_metadata_rewrite_inject_latency_ms=2 --log_container_max_size=16777216 It show that time spent about 77s before this patch, vs about 12s with this patch. -- To view, visit http://gerrit.cloudera.org:8080/17942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie48211d9e8c1d74e520fcb04df25c1d681261bb5 Gerrit-Change-Number: 17942 Gerrit-PatchSet: 3 Gerrit-Owner: Yingchun Lai <[email protected]> Gerrit-Reviewer: Abhishek Chennaka <[email protected]> Gerrit-Reviewer: Alexey Serbin <[email protected]> Gerrit-Reviewer: Andrew Wong <[email protected]> Gerrit-Reviewer: Greg Solovyev <[email protected]> Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Yingchun Lai <[email protected]> Gerrit-Comment-Date: Sat, 30 Oct 2021 05:18:43 +0000 Gerrit-HasComments: Yes
