Hello Kudu Jenkins, Andrew Wong, Grant Henke, Greg Solovyev, Bankim Bhavsar,
Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/15971
to look at the new patch set (#11).
Change subject: [master] cache for table locations
......................................................................
[master] cache for table locations
This patch introduces a cache for table locations in catalog manager.
When running with 48 concurrent client threads, the performance of
CatalogManager::GetTableLocations() method improved about 100%
when the cache is enabled.
A smaller 14% improvement is observed for GetTableLocations RPC
when running with the same number of concurrent client threads and cache
enabled. The test results are below.
I'm planning to add these test scenarios into kudu/scripts/benchmarks.sh
in a follow up patch.
========================================================================
After this patch with 128MByte cache enabled:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
--benchmark_num_threads=48 \
--table_locations_cache_capacity_mb=128
GetTableLocations function call: 504187.6 req/sec
Before this patch:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmarkFunctionCall \
--benchmark_num_threads=48
GetTableLocations function call: 252443.4 req/sec
========================================================================
After this patch with 128MByte cache enabled:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
--rpc_num_service_threads=32 \
--benchmark_num_threads=48 \
--table_locations_cache_capacity_mb=128
GetTableLocations RPC: 40033.4 req/sec
Stats on GetTableLocations RPC (times in microseconds):
Count: 200167
Mean: 155.144
Percentiles:
0% (min) = 45
25% = 125
50% (med) = 147
75% = 177
95% = 223
99% = 314
99.9% = 652
99.99% = 1928
100% (max) = 5096
Before this patch:
table_locations-itest \
--gtest_filter=TableLocationsTest.GetTableLocationsBenchmark \
--rpc_num_service_threads=32 \
--benchmark_num_threads=48
GetTableLocations RPC: 34981 req/sec
Stats on GetTableLocations RPC (times in microseconds):
Count: 174905
Mean: 249.308
Percentiles:
0% (min) = 64
25% = 197
50% (med) = 231
75% = 284
95% = 386
99% = 556
99.9% = 980
99.99% = 2400
100% (max) = 6113
Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
---
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/CMakeLists.txt
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master_service.cc
A src/kudu/master/table_locations_cache.cc
A src/kudu/master/table_locations_cache.h
A src/kudu/master/table_locations_cache_metrics.cc
A src/kudu/master/table_locations_cache_metrics.h
9 files changed, 1,098 insertions(+), 35 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/71/15971/11
--
To view, visit http://gerrit.cloudera.org:8080/15971
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7d2a4771ddc455d92a1da00db91c555a21151a23
Gerrit-Change-Number: 15971
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Bankim Bhavsar <ban...@cloudera.com>
Gerrit-Reviewer: Grant Henke <granthe...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gsolov...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
