Will Berkeley has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/11869 )
Change subject: [compaction] KUDU-1400: Improve rowset compaction policy to
consider merging small DRSs
......................................................................
[compaction] KUDU-1400: Improve rowset compaction policy to consider merging
small DRSs
This implements the small rowset compaction scheme explained in the
KUDU-1400 design doc [1]. While the implementation is simple, the
reasoning behind it is nuanced, and I'll defer the explanations of them
to the design doc rather than repeating them here.
The three relevant constant factors:
--compaction_minimum_improvement_score
--compaction_small_rowset_tradeoff
kSupportAdjust
have been tuned to work well with each other. See the relevant tests for
reasoning on why, and validation of what the results should be.
Since compaction policy's performance is important, I ran the YCSB
benchmark before and after the change using 'perf stat' to compare the
performance:
Command:
Before:
Performance counter stats for 'bin/compaction_policy-test
--gtest_filter=*Ycsb*' (10 runs):
1231.095442 task-clock # 0.997 CPUs utilized
( +- 2.00% )
148 context-switches # 0.120 K/sec
( +- 2.43% )
9 cpu-migrations # 0.007 K/sec
( +- 16.14% )
3,630 page-faults # 0.003 M/sec
( +- 0.00% )
3,251,530,478 cycles # 2.641 GHz
( +- 2.04% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
5,772,319,429 instructions # 1.78 insns per cycle
( +- 0.01% )
1,070,627,520 branches # 869.654 M/sec
( +- 0.01% )
13,583,368 branch-misses # 1.27% of all branches
( +- 0.10% )
1.235037947 seconds time elapsed
( +- 2.00% )
After:
Performance counter stats for 'bin/compaction_policy-test
--gtest_filter=*Ycsb*' (10 runs):
1297.749333 task-clock # 0.994 CPUs utilized
( +- 2.17% )
158 context-switches # 0.122 K/sec
( +- 3.01% )
14 cpu-migrations # 0.011 K/sec
( +- 17.48% )
3,636 page-faults # 0.003 M/sec
3,509,480,140 cycles # 2.704 GHz
( +- 2.65% )
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
6,800,316,126 instructions # 1.94 insns per cycle
( +- 0.01% )
1,073,111,416 branches # 826.902 M/sec
( +- 0.01% )
13,574,780 branch-misses # 1.26% of all branches
( +- 0.15% )
1.305058206 seconds time elapsed
( +- 2.16% )
A follow up will integrate the below design doc with the existing one,
docs/design-docs/compaction-policy.md.
[1]:
https://docs.google.com/document/d/1yTfxt0_2p5EfIjCnjJCt3o-nB9xk-Kl2O8yKTA1LQrQ/edit?usp=sharing
Change-Id: I7b421c6ed77d28ebab9b91a4d6fcb1e825997e6c
Reviewed-on: http://gerrit.cloudera.org:8080/11869
Tested-by: Kudu Jenkins
Reviewed-by: Andrew Wong <aw...@cloudera.com>
Reviewed-by: Alexey Serbin <aser...@cloudera.com>
---
M src/kudu/tablet/compaction_policy-test.cc
M src/kudu/tablet/compaction_policy.cc
M src/kudu/tablet/rowset_info.cc
M src/kudu/tablet/rowset_info.h
4 files changed, 297 insertions(+), 45 deletions(-)
Approvals:
Kudu Jenkins: Verified
Andrew Wong: Looks good to me, approved
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/11869
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I7b421c6ed77d28ebab9b91a4d6fcb1e825997e6c
Gerrit-Change-Number: 11869
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wdberke...@gmail.com>
Gerrit-Reviewer: Adar Dembo <a...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <aser...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Tidy Bot (241)
Gerrit-Reviewer: Todd Lipcon <t...@apache.org>
Gerrit-Reviewer: Will Berkeley <wdberke...@gmail.com>
