Todd Lipcon has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5512
Change subject: KUDU-1801: catalog_manager: change TableInfo lock to a rwlock
......................................................................
KUDU-1801: catalog_manager: change TableInfo lock to a rwlock
This lock was showing a lot of contention on a 200-node cluster with
40 concurrent Impala queries. This patch does a straightforward
substitution of a rw spinlock.
I deployed this on the cluster with the same workload and measured
metrics before and after:
Before:
{
"name": "handler_latency_kudu_master_MasterService_GetTableLocations",
"total_count": 10082475,
"min": 7,
"mean": 1229.28,
"percentile_75": 2128,
"percentile_95": 3456,
"percentile_99": 5056,
"percentile_99_9": 7680,
"percentile_99_99": 12928,
"max": 56026,
"total_sum": 12394196985
},
{
"name": "handler_latency_kudu_master_MasterService_GetTableSchema",
"total_count": 8372070,
"min": 18,
"mean": 2121.11,
"percentile_75": 2928,
"percentile_95": 4352,
"percentile_99": 6112,
"percentile_99_9": 9024,
"percentile_99_99": 16896,
"max": 67530,
"total_sum": 17758104544
}
After:
{
"name": "handler_latency_kudu_master_MasterService_GetTableLocations",
"total_count": 1231543,
"min": 2,
"mean": 86.0921,
"percentile_75": 101,
"percentile_95": 139,
"percentile_99": 176,
"percentile_99_9": 258,
"percentile_99_99": 868,
"max": 364560,
"total_sum": 106026108
},
{
"name": "handler_latency_kudu_master_MasterService_GetTableSchema",
"total_count": 1164715,
"min": 2,
"mean": 554.124,
"percentile_75": 620,
"percentile_95": 784,
"percentile_99": 1056,
"percentile_99_9": 1624,
"percentile_99_99": 6176,
"max": 364211,
"total_sum": 645396044
}
Improvement for GetTableLocations:
percentile_99_9: 29.8x
percentile_99_99: 14.9x
min: 3.5x
percentile_95: 24.9x
percentile_75: 21.1x
percentile_99: 28.7x
mean: 14.3x
Improvement for GetTableSchema:
percentile_99_9: 5.6x
percentile_99_99: 2.7x
min: 9.0x
percentile_95: 5.6x
percentile_75: 4.7x
percentile_99: 5.8x
mean: 3.8x
Change-Id: Id77edbf7641275b54010514379cdfff228b408b5
Reviewed-on: http://gerrit.cloudera.org:8080/5471
Reviewed-by: Adar Dembo <[email protected]>
Tested-by: Kudu Jenkins
(cherry picked from commit d59a9653a46d40c868c78fac6502e9a24e253731)
---
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
2 files changed, 16 insertions(+), 16 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/12/5512/1
--
To view, visit http://gerrit.cloudera.org:8080/5512
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Id77edbf7641275b54010514379cdfff228b408b5
Gerrit-PatchSet: 1
Gerrit-Project: kudu
Gerrit-Branch: branch-1.2.x
Gerrit-Owner: Todd Lipcon <[email protected]>
