[
https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364618#comment-17364618
]
Andrew Kyle Purtell edited comment on HBASE-25975 at 6/17/21, 1:09 AM:
-----------------------------------------------------------------------
Latest code with better microbenchmark, measuring exactly the time to call
region.batchMutate() at each iteration. Times are per op, measured in nanos,
converted to milliseconds for printing.
"0% contention case" -- All row keys in submitted requests are unique, so
should never overlap in the same clock tick. Differences in these values from
the baseline represent a combination of system variance and the additional
overheads introduced by the patch.
"100% contention case" -- All requests have the same duplicate set of row keys,
so should always overlap in the same clock tick. You can clearly see the
application of the constraint in the increases of MAX time, as expected.
Baseline:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0561
p99=0.0561, p999=0.0561 max=0.3001
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0552
p99=0.0564, p999=0.0564 max=1.0601
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0686
p99=0.0692, p999=0.0692 max=0.9828
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0792
p99=0.0813, p999=0.0813 max=1.3901
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1197
p99=0.1296, p999=0.1297 max=1.9159
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1318
p99=0.1414, p999=0.1415 max=4.8060
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0608
p99=0.0608, p999=0.0609 max=0.3793
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0533
p99=0.0559, p999=0.0559 max=0.4041
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0603
p99=0.0612, p999=0.0612 max=0.4097
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1045
p99=0.1121, p999=0.1121 max=0.8509
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1334
p99=0.1422, p999=0.1426 max=1.2274
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1301
p99=0.1399, p999=0.1400 max=2.9175
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1789
p99=0.1789, p999=0.1789 max=0.4747
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1709
p99=0.1742, p999=0.1743 max=0.6043
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1820
p99=0.1897, p999=0.1898 max=2.5493
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2809
p99=0.2856, p999=0.2857 max=4.1059
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.4268
p99=0.4393, p999=0.4394 max=5.5858
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6382
p99=0.7335, p999=0.7338 max=16.4132
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.5447
p99=1.5447, p999=1.5448 max=2.4460
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6133
p99=1.6489, p999=1.6493 max=10.4490
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1818
p99=2.2960, p999=2.2995 max=23.4926
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.3715
p99=2.4573, p999=2.4615 max=33.7535
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3571
p99=4.4556, p999=4.4600 max=111.8534
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.4922
p99=5.4098, p999=5.4190 max=225.4612
{noformat}
With RowCommitSequencer 0% contention case:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1108
p99=0.1108, p999=0.1108 max=3.3199
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0902
p99=0.1213, p999=0.1216 max=1.7926
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1287
p99=0.1346, p999=0.1347 max=1.8199
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1200
p99=0.1365, p999=0.1367 max=1.9109
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1298
p99=0.1352, p999=0.1352 max=1.9318
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1315
p99=0.1522, p999=0.1525 max=2.6332
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0595
p99=0.0595, p999=0.0596 max=1.6384
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0684
p99=0.0692, p999=0.0692 max=1.4759
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0918
p99=0.0973, p999=0.0974 max=1.6548
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0929
p99=0.1042, p999=0.1043 max=1.7982
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1206
p99=0.1313, p999=0.1313 max=1.7856
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1198
p99=0.1400, p999=0.1400 max=4.4632
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2106
p99=0.2106, p999=0.2106 max=2.2026
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1910
p99=0.2283, p999=0.2287 max=2.0957
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2774
p99=0.2824, p999=0.2824 max=3.6852
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3029
p99=0.3219, p999=0.3224 max=4.0838
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3813
p99=0.4126, p999=0.4129 max=7.7517
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6552
p99=0.7831, p999=0.7837 max=16.8252
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.4204
p99=1.4204, p999=1.4209 max=17.4774
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6563
p99=1.6654, p999=1.6662 max=22.6109
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1168
p99=2.1457, p999=2.1463 max=22.9576
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.4217
p99=2.6266, p999=2.6272 max=28.9599
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3466
p99=4.4730, p999=4.4732 max=126.3769
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=5.3735
p99=5.7779, p999=5.7862 max=233.5676
{noformat}
With RowCommitSequencer 100% contention case:
{noformat}
1 threads 1 contended rows 100 iterations, ms/op: p50=0.0558 p99=0.0558,
p999=0.0562 max=1.7508
2 threads 1 contended rows 100 iterations, ms/op: p50=0.0683 p99=0.0808,
p999=0.0810 max=4.3242
4 threads 1 contended rows 100 iterations, ms/op: p50=0.0946 p99=0.1012,
p999=0.1013 max=16.4388
8 threads 1 contended rows 100 iterations, ms/op: p50=0.1177 p99=0.1249,
p999=0.1251 max=32.6156
16 threads 1 contended rows 100 iterations, ms/op: p50=0.1444 p99=0.1591,
p999=0.1596 max=91.9189
32 threads 1 contended rows 100 iterations, ms/op: p50=0.1805 p99=1.2789,
p999=1.2793 max=230.0010
1 threads 10 contended rows 100 iterations, ms/op: p50=0.0756 p99=0.0756,
p999=0.0757 max=1.6382
2 threads 10 contended rows 100 iterations, ms/op: p50=0.0800 p99=0.0837,
p999=0.0838 max=5.7082
4 threads 10 contended rows 100 iterations, ms/op: p50=0.0972 p99=0.1088,
p999=0.1090 max=13.2040
8 threads 10 contended rows 100 iterations, ms/op: p50=0.1042 p99=0.1423,
p999=0.1431 max=28.6613
16 threads 10 contended rows 100 iterations, ms/op: p50=0.1571 p99=0.1633,
p999=0.1633 max=102.7910
32 threads 10 contended rows 100 iterations, ms/op: p50=0.1630 p99=1.2682,
p999=1.2702 max=248.9046
1 threads 100 contended rows 100 iterations, ms/op: p50=0.2042 p99=0.2042,
p999=0.2044 max=2.1793
2 threads 100 contended rows 100 iterations, ms/op: p50=0.2054 p99=0.2342,
p999=0.2348 max=4.2469
4 threads 100 contended rows 100 iterations, ms/op: p50=0.2488 p99=0.2831,
p999=0.2838 max=17.7257
8 threads 100 contended rows 100 iterations, ms/op: p50=0.2649 p99=0.3028,
p999=0.3029 max=44.8713
16 threads 100 contended rows 100 iterations, ms/op: p50=0.2779 p99=0.3044,
p999=0.3049 max=99.4218
32 threads 100 contended rows 100 iterations, ms/op: p50=0.3135 p99=0.3263,
p999=0.3264 max=355.3952
1 threads 1000 contended rows 100 iterations, ms/op: p50=1.5648 p99=1.5648,
p999=1.5649 max=2.6714
2 threads 1000 contended rows 100 iterations, ms/op: p50=1.5051 p99=1.6097,
p999=1.6111 max=83.0051
4 threads 1000 contended rows 100 iterations, ms/op: p50=1.8555 p99=1.8793,
p999=1.8794 max=20.8204
8 threads 1000 contended rows 100 iterations, ms/op: p50=1.8331 p99=1.9490,
p999=1.9494 max=33.7916
16 threads 1000 contended rows 100 iterations, ms/op: p50=1.9460 p99=2.0191,
p999=2.0193 max=187.0420
32 threads 1000 contended rows 100 iterations, ms/op: p50=2.4060 p99=2.6065,
p999=2.6118 max=302.2282
{noformat}
was (Author: apurtell):
Latest code with better microbenchmark, measuring exactly the time to call
region.batchMutate() at each iteration. Times are per op, measured in nanos,
converted to milliseconds for printing.
"0% contention case" -- All row keys in submitted requests are unique, so
should never overlap in the same clock tick.
"100% contention case" -- All requests have the same duplicate set of row keys,
so should always overlap in the same clock tick.
Baseline:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0561
p99=0.0561, p999=0.0561 max=0.3001
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0552
p99=0.0564, p999=0.0564 max=1.0601
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0686
p99=0.0692, p999=0.0692 max=0.9828
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0792
p99=0.0813, p999=0.0813 max=1.3901
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1197
p99=0.1296, p999=0.1297 max=1.9159
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1318
p99=0.1414, p999=0.1415 max=4.8060
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0608
p99=0.0608, p999=0.0609 max=0.3793
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0533
p99=0.0559, p999=0.0559 max=0.4041
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0603
p99=0.0612, p999=0.0612 max=0.4097
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1045
p99=0.1121, p999=0.1121 max=0.8509
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1334
p99=0.1422, p999=0.1426 max=1.2274
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1301
p99=0.1399, p999=0.1400 max=2.9175
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1789
p99=0.1789, p999=0.1789 max=0.4747
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1709
p99=0.1742, p999=0.1743 max=0.6043
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1820
p99=0.1897, p999=0.1898 max=2.5493
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2809
p99=0.2856, p999=0.2857 max=4.1059
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.4268
p99=0.4393, p999=0.4394 max=5.5858
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6382
p99=0.7335, p999=0.7338 max=16.4132
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.5447
p99=1.5447, p999=1.5448 max=2.4460
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6133
p99=1.6489, p999=1.6493 max=10.4490
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1818
p99=2.2960, p999=2.2995 max=23.4926
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.3715
p99=2.4573, p999=2.4615 max=33.7535
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3571
p99=4.4556, p999=4.4600 max=111.8534
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.4922
p99=5.4098, p999=5.4190 max=225.4612
{noformat}
With RowCommitSequencer 0% contention case:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1108
p99=0.1108, p999=0.1108 max=3.3199
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0902
p99=0.1213, p999=0.1216 max=1.7926
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1287
p99=0.1346, p999=0.1347 max=1.8199
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1200
p99=0.1365, p999=0.1367 max=1.9109
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1298
p99=0.1352, p999=0.1352 max=1.9318
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1315
p99=0.1522, p999=0.1525 max=2.6332
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0595
p99=0.0595, p999=0.0596 max=1.6384
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0684
p99=0.0692, p999=0.0692 max=1.4759
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0918
p99=0.0973, p999=0.0974 max=1.6548
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0929
p99=0.1042, p999=0.1043 max=1.7982
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1206
p99=0.1313, p999=0.1313 max=1.7856
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1198
p99=0.1400, p999=0.1400 max=4.4632
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2106
p99=0.2106, p999=0.2106 max=2.2026
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1910
p99=0.2283, p999=0.2287 max=2.0957
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2774
p99=0.2824, p999=0.2824 max=3.6852
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3029
p99=0.3219, p999=0.3224 max=4.0838
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3813
p99=0.4126, p999=0.4129 max=7.7517
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6552
p99=0.7831, p999=0.7837 max=16.8252
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.4204
p99=1.4204, p999=1.4209 max=17.4774
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6563
p99=1.6654, p999=1.6662 max=22.6109
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1168
p99=2.1457, p999=2.1463 max=22.9576
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.4217
p99=2.6266, p999=2.6272 max=28.9599
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3466
p99=4.4730, p999=4.4732 max=126.3769
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=5.3735
p99=5.7779, p999=5.7862 max=233.5676
{noformat}
With RowCommitSequencer 100% contention case:
{noformat}
1 threads 1 contended rows 100 iterations, ms/op: p50=0.0558 p99=0.0558,
p999=0.0562 max=1.7508
2 threads 1 contended rows 100 iterations, ms/op: p50=0.0683 p99=0.0808,
p999=0.0810 max=4.3242
4 threads 1 contended rows 100 iterations, ms/op: p50=0.0946 p99=0.1012,
p999=0.1013 max=16.4388
8 threads 1 contended rows 100 iterations, ms/op: p50=0.1177 p99=0.1249,
p999=0.1251 max=32.6156
16 threads 1 contended rows 100 iterations, ms/op: p50=0.1444 p99=0.1591,
p999=0.1596 max=91.9189
32 threads 1 contended rows 100 iterations, ms/op: p50=0.1805 p99=1.2789,
p999=1.2793 max=230.0010
1 threads 10 contended rows 100 iterations, ms/op: p50=0.0756 p99=0.0756,
p999=0.0757 max=1.6382
2 threads 10 contended rows 100 iterations, ms/op: p50=0.0800 p99=0.0837,
p999=0.0838 max=5.7082
4 threads 10 contended rows 100 iterations, ms/op: p50=0.0972 p99=0.1088,
p999=0.1090 max=13.2040
8 threads 10 contended rows 100 iterations, ms/op: p50=0.1042 p99=0.1423,
p999=0.1431 max=28.6613
16 threads 10 contended rows 100 iterations, ms/op: p50=0.1571 p99=0.1633,
p999=0.1633 max=102.7910
32 threads 10 contended rows 100 iterations, ms/op: p50=0.1630 p99=1.2682,
p999=1.2702 max=248.9046
1 threads 100 contended rows 100 iterations, ms/op: p50=0.2042 p99=0.2042,
p999=0.2044 max=2.1793
2 threads 100 contended rows 100 iterations, ms/op: p50=0.2054 p99=0.2342,
p999=0.2348 max=4.2469
4 threads 100 contended rows 100 iterations, ms/op: p50=0.2488 p99=0.2831,
p999=0.2838 max=17.7257
8 threads 100 contended rows 100 iterations, ms/op: p50=0.2649 p99=0.3028,
p999=0.3029 max=44.8713
16 threads 100 contended rows 100 iterations, ms/op: p50=0.2779 p99=0.3044,
p999=0.3049 max=99.4218
32 threads 100 contended rows 100 iterations, ms/op: p50=0.3135 p99=0.3263,
p999=0.3264 max=355.3952
1 threads 1000 contended rows 100 iterations, ms/op: p50=1.5648 p99=1.5648,
p999=1.5649 max=2.6714
2 threads 1000 contended rows 100 iterations, ms/op: p50=1.5051 p99=1.6097,
p999=1.6111 max=83.0051
4 threads 1000 contended rows 100 iterations, ms/op: p50=1.8555 p99=1.8793,
p999=1.8794 max=20.8204
8 threads 1000 contended rows 100 iterations, ms/op: p50=1.8331 p99=1.9490,
p999=1.9494 max=33.7916
16 threads 1000 contended rows 100 iterations, ms/op: p50=1.9460 p99=2.0191,
p999=2.0193 max=187.0420
32 threads 1000 contended rows 100 iterations, ms/op: p50=2.4060 p99=2.6065,
p999=2.6118 max=302.2282
{noformat}
> Row commit sequencer
> --------------------
>
> Key: HBASE-25975
> URL: https://issues.apache.org/jira/browse/HBASE-25975
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: HBASE-25975-c4cf83ce.pdf
>
>
> Use a row commit sequencer in HRegion to ensure that only the operations that
> mutate disjoint sets of rows are able to commit within the same clock tick.
> This maintains the invariant that more than one mutation to a given row will
> never be committed in the same clock tick.
> Callers will first acquire row locks for the row(s) the pending mutation will
> mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that
> the set of rows about to be mutated do not overlap with those for any other
> pending mutations in the current clock tick. If an overlap is identified,
> getRowSequence will yield and loop until there is no longer an overlap and
> the caller's pending mutation can succeed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
