[
https://issues.apache.org/jira/browse/HBASE-25975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17364618#comment-17364618
]
Andrew Kyle Purtell edited comment on HBASE-25975 at 6/17/21, 2:16 AM:
-----------------------------------------------------------------------
Latest code with better microbenchmark, measuring exactly the time to call
region.batchMutate() at each iteration. Times are per op, measured in nanos,
converted to milliseconds for printing.
"0% contention case" -- All row keys in submitted requests are unique, so
should never overlap in the same clock tick. Differences in these values from
the baseline represent a combination of system variance and the additional
overheads introduced by the patch.
"100% contention case" -- All requests have the same duplicate set of row keys,
so should always overlap in the same clock tick. You can clearly see the
application of the constraint in the increases of MAX time, as expected.
Baseline:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0561
p99=0.0561, p999=0.0561 max=0.3001
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0552
p99=0.0564, p999=0.0564 max=1.0601
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0686
p99=0.0692, p999=0.0692 max=0.9828
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0792
p99=0.0813, p999=0.0813 max=1.3901
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1197
p99=0.1296, p999=0.1297 max=1.9159
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1318
p99=0.1414, p999=0.1415 max=4.8060
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0608
p99=0.0608, p999=0.0609 max=0.3793
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0533
p99=0.0559, p999=0.0559 max=0.4041
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0603
p99=0.0612, p999=0.0612 max=0.4097
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1045
p99=0.1121, p999=0.1121 max=0.8509
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1334
p99=0.1422, p999=0.1426 max=1.2274
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1301
p99=0.1399, p999=0.1400 max=2.9175
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1789
p99=0.1789, p999=0.1789 max=0.4747
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1709
p99=0.1742, p999=0.1743 max=0.6043
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1820
p99=0.1897, p999=0.1898 max=2.5493
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2809
p99=0.2856, p999=0.2857 max=4.1059
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.4268
p99=0.4393, p999=0.4394 max=5.5858
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6382
p99=0.7335, p999=0.7338 max=16.4132
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.5447
p99=1.5447, p999=1.5448 max=2.4460
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6133
p99=1.6489, p999=1.6493 max=10.4490
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1818
p99=2.2960, p999=2.2995 max=23.4926
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.3715
p99=2.4573, p999=2.4615 max=33.7535
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3571
p99=4.4556, p999=4.4600 max=111.8534
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.4922
p99=5.4098, p999=5.4190 max=225.4612
{noformat}
With RowCommitSequencer 0% contention case:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1207
p99=0.1207, p999=0.1209 max=1.8813
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1048
p99=0.1102, p999=0.1103 max=2.1397
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1166
p99=0.1312, p999=0.1315 max=1.7879
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1331
p99=0.1440, p999=0.1443 max=3.7075
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1464
p99=0.1697, p999=0.1704 max=1.9381
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1386
p99=0.1524, p999=0.1525 max=2.3399
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0690
p99=0.0690, p999=0.0691 max=1.5870
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1142
p99=0.1170, p999=0.1171 max=1.7043
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0944
p99=0.0959, p999=0.0959 max=1.7438
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1086
p99=0.1111, p999=0.1111 max=1.9832
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1259
p99=0.1396, p999=0.1398 max=1.7491
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1155
p99=0.1261, p999=0.1264 max=4.7331
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1732
p99=0.1732, p999=0.1733 max=1.6654
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2309
p99=0.2325, p999=0.2326 max=1.8260
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2475
p99=0.2778, p999=0.2785 max=3.4691
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2914
p99=0.3106, p999=0.3107 max=5.4343
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.4262
p99=0.4465, p999=0.4472 max=11.6418
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6965
p99=0.7206, p999=0.7213 max=15.0728
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.4351
p99=1.4351, p999=1.4358 max=15.4978
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6366
p99=1.6577, p999=1.6582 max=4.2604
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.9030
p99=2.0894, p999=2.0909 max=21.2956
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.5048
p99=2.6549, p999=2.6581 max=33.1727
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.4060
p99=4.8365, p999=4.8433 max=108.4194
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.7558
p99=5.9772, p999=5.9816 max=175.4064
{noformat}
With RowCommitSequencer 100% contention case:
{noformat}
1 threads 1 contended rows 100 iterations, ms/op: p50=0.0582 p99=0.0582,
p999=0.0584 max=3.0230
2 threads 1 contended rows 100 iterations, ms/op: p50=0.0812 p99=0.0825,
p999=0.0825 max=4.1251
4 threads 1 contended rows 100 iterations, ms/op: p50=0.0884 p99=0.1229,
p999=0.1230 max=13.6791
8 threads 1 contended rows 100 iterations, ms/op: p50=0.1205 p99=0.1260,
p999=0.1260 max=31.4853
16 threads 1 contended rows 100 iterations, ms/op: p50=0.1221 p99=0.1411,
p999=0.1414 max=115.6539
32 threads 1 contended rows 100 iterations, ms/op: p50=0.1067 p99=0.1659,
p999=0.1691 max=233.7836
1 threads 10 contended rows 100 iterations, ms/op: p50=0.1023 p99=0.1023,
p999=0.1024 max=1.6838
2 threads 10 contended rows 100 iterations, ms/op: p50=0.0663 p99=0.0701,
p999=0.0702 max=5.3990
4 threads 10 contended rows 100 iterations, ms/op: p50=0.0837 p99=0.1030,
p999=0.1033 max=10.6789
8 threads 10 contended rows 100 iterations, ms/op: p50=0.0928 p99=0.1494,
p999=0.1496 max=32.4364
16 threads 10 contended rows 100 iterations, ms/op: p50=0.1625 p99=0.1759,
p999=0.1762 max=133.6848
32 threads 10 contended rows 100 iterations, ms/op: p50=0.1650 p99=0.1946,
p999=0.1948 max=245.6805
1 threads 100 contended rows 100 iterations, ms/op: p50=0.2607 p99=0.2607,
p999=0.2608 max=2.3032
2 threads 100 contended rows 100 iterations, ms/op: p50=0.2449 p99=0.2583,
p999=0.2584 max=4.0113
4 threads 100 contended rows 100 iterations, ms/op: p50=0.2803 p99=0.3129,
p999=0.3129 max=104.3184
8 threads 100 contended rows 100 iterations, ms/op: p50=0.2778 p99=0.3147,
p999=0.3150 max=42.7154
16 threads 100 contended rows 100 iterations, ms/op: p50=0.3228 p99=0.3680,
p999=0.3685 max=105.5341
32 threads 100 contended rows 100 iterations, ms/op: p50=0.3124 p99=0.3333,
p999=0.3334 max=222.9051
1 threads 1000 contended rows 100 iterations, ms/op: p50=1.5311 p99=1.5311,
p999=1.5314 max=2.3854
2 threads 1000 contended rows 100 iterations, ms/op: p50=1.2614 p99=1.2890,
p999=1.2893 max=6.5509
4 threads 1000 contended rows 100 iterations, ms/op: p50=1.8523 p99=1.9340,
p999=1.9349 max=22.0056
8 threads 1000 contended rows 100 iterations, ms/op: p50=1.8517 p99=1.9559,
p999=1.9563 max=36.0947
16 threads 1000 contended rows 100 iterations, ms/op: p50=1.9385 p99=1.9811,
p999=1.9815 max=124.6219
32 threads 1000 contended rows 100 iterations, ms/op: p50=2.3027 p99=2.8692,
p999=2.8748 max=270.6748
{noformat}
was (Author: apurtell):
Latest code with better microbenchmark, measuring exactly the time to call
region.batchMutate() at each iteration. Times are per op, measured in nanos,
converted to milliseconds for printing.
"0% contention case" -- All row keys in submitted requests are unique, so
should never overlap in the same clock tick. Differences in these values from
the baseline represent a combination of system variance and the additional
overheads introduced by the patch.
"100% contention case" -- All requests have the same duplicate set of row keys,
so should always overlap in the same clock tick. You can clearly see the
application of the constraint in the increases of MAX time, as expected.
Baseline:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0561
p99=0.0561, p999=0.0561 max=0.3001
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0552
p99=0.0564, p999=0.0564 max=1.0601
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0686
p99=0.0692, p999=0.0692 max=0.9828
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0792
p99=0.0813, p999=0.0813 max=1.3901
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1197
p99=0.1296, p999=0.1297 max=1.9159
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1318
p99=0.1414, p999=0.1415 max=4.8060
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0608
p99=0.0608, p999=0.0609 max=0.3793
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0533
p99=0.0559, p999=0.0559 max=0.4041
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0603
p99=0.0612, p999=0.0612 max=0.4097
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1045
p99=0.1121, p999=0.1121 max=0.8509
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1334
p99=0.1422, p999=0.1426 max=1.2274
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1301
p99=0.1399, p999=0.1400 max=2.9175
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1789
p99=0.1789, p999=0.1789 max=0.4747
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1709
p99=0.1742, p999=0.1743 max=0.6043
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1820
p99=0.1897, p999=0.1898 max=2.5493
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2809
p99=0.2856, p999=0.2857 max=4.1059
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.4268
p99=0.4393, p999=0.4394 max=5.5858
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6382
p99=0.7335, p999=0.7338 max=16.4132
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.5447
p99=1.5447, p999=1.5448 max=2.4460
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6133
p99=1.6489, p999=1.6493 max=10.4490
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1818
p99=2.2960, p999=2.2995 max=23.4926
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.3715
p99=2.4573, p999=2.4615 max=33.7535
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3571
p99=4.4556, p999=4.4600 max=111.8534
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.4922
p99=5.4098, p999=5.4190 max=225.4612
{noformat}
With RowCommitSequencer 0% contention case:
{noformat}
1 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1108
p99=0.1108, p999=0.1108 max=3.3199
2 threads 1 non-contended rows 100 iterations, ms/op: p50=0.0902
p99=0.1213, p999=0.1216 max=1.7926
4 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1287
p99=0.1346, p999=0.1347 max=1.8199
8 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1200
p99=0.1365, p999=0.1367 max=1.9109
16 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1298
p99=0.1352, p999=0.1352 max=1.9318
32 threads 1 non-contended rows 100 iterations, ms/op: p50=0.1315
p99=0.1522, p999=0.1525 max=2.6332
1 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0595
p99=0.0595, p999=0.0596 max=1.6384
2 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0684
p99=0.0692, p999=0.0692 max=1.4759
4 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0918
p99=0.0973, p999=0.0974 max=1.6548
8 threads 10 non-contended rows 100 iterations, ms/op: p50=0.0929
p99=0.1042, p999=0.1043 max=1.7982
16 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1206
p99=0.1313, p999=0.1313 max=1.7856
32 threads 10 non-contended rows 100 iterations, ms/op: p50=0.1198
p99=0.1400, p999=0.1400 max=4.4632
1 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2106
p99=0.2106, p999=0.2106 max=2.2026
2 threads 100 non-contended rows 100 iterations, ms/op: p50=0.1910
p99=0.2283, p999=0.2287 max=2.0957
4 threads 100 non-contended rows 100 iterations, ms/op: p50=0.2774
p99=0.2824, p999=0.2824 max=3.6852
8 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3029
p99=0.3219, p999=0.3224 max=4.0838
16 threads 100 non-contended rows 100 iterations, ms/op: p50=0.3813
p99=0.4126, p999=0.4129 max=7.7517
32 threads 100 non-contended rows 100 iterations, ms/op: p50=0.6552
p99=0.7831, p999=0.7837 max=16.8252
1 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.4204
p99=1.4204, p999=1.4209 max=17.4774
2 threads 1000 non-contended rows 100 iterations, ms/op: p50=1.6563
p99=1.6654, p999=1.6662 max=22.6109
4 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.1168
p99=2.1457, p999=2.1463 max=22.9576
8 threads 1000 non-contended rows 100 iterations, ms/op: p50=2.4217
p99=2.6266, p999=2.6272 max=28.9599
16 threads 1000 non-contended rows 100 iterations, ms/op: p50=4.3466
p99=4.4730, p999=4.4732 max=126.3769
32 threads 1000 non-contended rows 100 iterations, ms/op: p50=5.3735
p99=5.7779, p999=5.7862 max=233.5676
{noformat}
With RowCommitSequencer 100% contention case:
{noformat}
1 threads 1 contended rows 100 iterations, ms/op: p50=0.0558 p99=0.0558,
p999=0.0562 max=1.7508
2 threads 1 contended rows 100 iterations, ms/op: p50=0.0683 p99=0.0808,
p999=0.0810 max=4.3242
4 threads 1 contended rows 100 iterations, ms/op: p50=0.0946 p99=0.1012,
p999=0.1013 max=16.4388
8 threads 1 contended rows 100 iterations, ms/op: p50=0.1177 p99=0.1249,
p999=0.1251 max=32.6156
16 threads 1 contended rows 100 iterations, ms/op: p50=0.1444 p99=0.1591,
p999=0.1596 max=91.9189
32 threads 1 contended rows 100 iterations, ms/op: p50=0.1805 p99=1.2789,
p999=1.2793 max=230.0010
1 threads 10 contended rows 100 iterations, ms/op: p50=0.0756 p99=0.0756,
p999=0.0757 max=1.6382
2 threads 10 contended rows 100 iterations, ms/op: p50=0.0800 p99=0.0837,
p999=0.0838 max=5.7082
4 threads 10 contended rows 100 iterations, ms/op: p50=0.0972 p99=0.1088,
p999=0.1090 max=13.2040
8 threads 10 contended rows 100 iterations, ms/op: p50=0.1042 p99=0.1423,
p999=0.1431 max=28.6613
16 threads 10 contended rows 100 iterations, ms/op: p50=0.1571 p99=0.1633,
p999=0.1633 max=102.7910
32 threads 10 contended rows 100 iterations, ms/op: p50=0.1630 p99=1.2682,
p999=1.2702 max=248.9046
1 threads 100 contended rows 100 iterations, ms/op: p50=0.2042 p99=0.2042,
p999=0.2044 max=2.1793
2 threads 100 contended rows 100 iterations, ms/op: p50=0.2054 p99=0.2342,
p999=0.2348 max=4.2469
4 threads 100 contended rows 100 iterations, ms/op: p50=0.2488 p99=0.2831,
p999=0.2838 max=17.7257
8 threads 100 contended rows 100 iterations, ms/op: p50=0.2649 p99=0.3028,
p999=0.3029 max=44.8713
16 threads 100 contended rows 100 iterations, ms/op: p50=0.2779 p99=0.3044,
p999=0.3049 max=99.4218
32 threads 100 contended rows 100 iterations, ms/op: p50=0.3135 p99=0.3263,
p999=0.3264 max=355.3952
1 threads 1000 contended rows 100 iterations, ms/op: p50=1.5648 p99=1.5648,
p999=1.5649 max=2.6714
2 threads 1000 contended rows 100 iterations, ms/op: p50=1.5051 p99=1.6097,
p999=1.6111 max=83.0051
4 threads 1000 contended rows 100 iterations, ms/op: p50=1.8555 p99=1.8793,
p999=1.8794 max=20.8204
8 threads 1000 contended rows 100 iterations, ms/op: p50=1.8331 p99=1.9490,
p999=1.9494 max=33.7916
16 threads 1000 contended rows 100 iterations, ms/op: p50=1.9460 p99=2.0191,
p999=2.0193 max=187.0420
32 threads 1000 contended rows 100 iterations, ms/op: p50=2.4060 p99=2.6065,
p999=2.6118 max=302.2282
{noformat}
> Row commit sequencer
> --------------------
>
> Key: HBASE-25975
> URL: https://issues.apache.org/jira/browse/HBASE-25975
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
> Reporter: Andrew Kyle Purtell
> Assignee: Andrew Kyle Purtell
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: HBASE-25975-c4cf83ce.pdf
>
>
> Use a row commit sequencer in HRegion to ensure that only the operations that
> mutate disjoint sets of rows are able to commit within the same clock tick.
> This maintains the invariant that more than one mutation to a given row will
> never be committed in the same clock tick.
> Callers will first acquire row locks for the row(s) the pending mutation will
> mutate. Then they will use RowCommitSequencer.getRowSequence to ensure that
> the set of rows about to be mutated do not overlap with those for any other
> pending mutations in the current clock tick. If an overlap is identified,
> getRowSequence will yield and loop until there is no longer an overlap and
> the caller's pending mutation can succeed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
