[
https://issues.apache.org/jira/browse/HBASE-6295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644489#comment-13644489
]
Nicolas Liochon commented on HBASE-6295:
----------------------------------------
I've changed the per server task limit to a per region task limit.
Here is the result of a test on 5 datanodes, ec2 xlarge instances, table not
split. I've executed the test multiple times on each config.
{code}
./bin/ycsb load hbase -P workloads/workloada -s -p columnfamily=family -p
recordcount=100000000 | grep -v CLEAN | grep -v INSERT | grep -v UPDATE
{code}
without 6295
{noformat}
10 sec: 50159 operations; 5011,39 current ops/sec; [INSERT
AverageLatency(us)=184,77]
20 sec: 127679 operations; 7748,9 current ops/sec; [INSERT
AverageLatency(us)=121,76]
30 sec: 132239 operations; 455,77 current ops/sec; [INSERT
AverageLatency(us)=242,47]
40 sec: 132239 operations; 0 current ops/sec;
50 sec: 132239 operations; 0 current ops/sec;
60 sec: 132239 operations; 0 current ops/sec;
70 sec: 132239 operations; 0 current ops/sec;
80 sec: 132239 operations; 0 current ops/sec;
90 sec: 167676 operations; 3532,04 current ops/sec; [INSERT
AverageLatency(us)=1955,36]
100 sec: 250799 operations; 8308,98 current ops/sec; [INSERT
AverageLatency(us)=114,35]
110 sec: 341999 operations; 9115,44 current ops/sec; [INSERT
AverageLatency(us)=107,25]
120 sec: 428639 operations; 8660,54 current ops/sec; [INSERT
AverageLatency(us)=109,67]
130 sec: 507844 operations; 7916,54 current ops/sec; [INSERT
AverageLatency(us)=125,44]
140 sec: 588239 operations; 8035,48 current ops/sec; [INSERT
AverageLatency(us)=120,31]
150 sec: 674879 operations; 8660,54 current ops/sec; [INSERT
AverageLatency(us)=109,59]
160 sec: 729599 operations; 5469,27 current ops/sec; [INSERT
AverageLatency(us)=111,25]
170 sec: 729599 operations; 0 current ops/sec;
180 sec: 729599 operations; 0 current ops/sec;
190 sec: 729599 operations; 0 current ops/sec;
200 sec: 729599 operations; 0 current ops/sec;
210 sec: 729599 operations; 0 current ops/sec;
220 sec: 729599 operations; 0 current ops/sec;
230 sec: 819076 operations; 8926,28 current ops/sec; [INSERT
AverageLatency(us)=824,06]
240 sec: 911999 operations; 9287,66 current ops/sec; [INSERT
AverageLatency(us)=102,68]
250 sec: 998639 operations; 8659,67 current ops/sec; [INSERT
AverageLatency(us)=109,97]
260 sec: 1089839 operations; 9116,35 current ops/sec; [INSERT
AverageLatency(us)=105,23]
270 sec: 1094399 operations; 455,77 current ops/sec; [INSERT
AverageLatency(us)=219,31]
280 sec: 1094399 operations; 0 current ops/sec;
290 sec: 1094399 operations; 0 current ops/sec;
300 sec: 1098959 operations; 455,77 current ops/sec; [INSERT
AverageLatency(us)=7326,11]
310 sec: 1098959 operations; 0 current ops/sec;
320 sec: 1098959 operations; 0 current ops/sec;
330 sec: 1131203 operations; 3222,79 current ops/sec; [INSERT
AverageLatency(us)=1109,09]
2013-04-29 09:06:04,357 INFO [Thread-1] zookeeper.RecoverableZooKeeper
(RecoverableZooKeeper.java:<init>(119)) - The identifier of this process is
hconnection-0x8888e6c
2013-04-29 09:06:04,373 INFO
[Thread-1-SendThread(ip-10-68-135-50.ec2.internal:2181)] zookeeper.ClientCnxn
(ClientCnxn.java:logStartConnect(966)) - Opening socket connection to server
ip-10-68-135-50.ec2.internal/10.68.135.50:2181. Will not attempt to
authenticate using SASL (Impossible de trouver une configuration de connexion)
2013-04-29 09:06:04,380 INFO
[Thread-1-SendThread(ip-10-68-135-50.ec2.internal:2181)] zookeeper.ClientCnxn
(ClientCnxn.java:primeConnection(849)) - Socket connection established to
ip-10-68-135-50.ec2.internal/10.68.135.50:2181, initiating session
2013-04-29 09:06:04,391 INFO
[Thread-1-SendThread(ip-10-68-135-50.ec2.internal:2181)] zookeeper.ClientCnxn
(ClientCnxn.java:onConnected(1207)) - Session establishment complete on server
ip-10-68-135-50.ec2.internal/10.68.135.50:2181, sessionid = 0x13e55e55e6e000e,
negotiated timeout = 180000
340 sec: 1158239 operations; 2702,52 current ops/sec; [INSERT
AverageLatency(us)=39,33]
350 sec: 1158239 operations; 0 current ops/sec;
360 sec: 1158239 operations; 0 current ops/sec;
370 sec: 1158239 operations; 0 current ops/sec;
380 sec: 1158239 operations; 0 current ops/sec;
390 sec: 1158239 operations; 0 current ops/sec;
400 sec: 1244879 operations; 8659,67 current ops/sec; [INSERT
AverageLatency(us)=790,06]
410 sec: 1326959 operations; 8203,9 current ops/sec; [INSERT
AverageLatency(us)=52,75]
420 sec: 1326959 operations; 0 current ops/sec;
430 sec: 1326959 operations; 0 current ops/sec;
440 sec: 1326959 operations; 0 current ops/sec;
450 sec: 1326959 operations; 0 current ops/sec;
460 sec: 1326959 operations; 0 current ops/sec;
470 sec: 1349759 operations; 2279,09 current ops/sec; [INSERT
AverageLatency(us)=2868,24]
480 sec: 1527599 operations; 17775,11 current ops/sec; [INSERT
AverageLatency(us)=51,74]
490 sec: 1723679 operations; 19535,72 current ops/sec; [INSERT
AverageLatency(us)=47,81]
500 sec: 1924319 operations; 20015,96 current ops/sec; [INSERT
AverageLatency(us)=45,59]
510 sec: 2129519 operations; 20509,75 current ops/sec; [INSERT
AverageLatency(us)=44,74]
520 sec: 2307359 operations; 17776,89 current ops/sec; [INSERT
AverageLatency(us)=52,51]
530 sec: 2476079 operations; 16813,15 current ops/sec; [INSERT
AverageLatency(us)=55,7]
540 sec: 2647330 operations; 17099,45 current ops/sec; [INSERT
AverageLatency(us)=54,69]
550 sec: 2831759 operations; 18435,53 current ops/sec; [INSERT
AverageLatency(us)=45,4]
560 sec: 2831759 operations; 0 current ops/sec;
570 sec: 2831759 operations; 0 current ops/sec;
580 sec: 2831759 operations; 0 current ops/sec;
590 sec: 2831759 operations; 0 current ops/sec;
600 sec: 2831759 operations; 0 current ops/sec;
610 sec: 2831759 operations; 0 current ops/sec;
620 sec: 2963999 operations; 13217,39 current ops/sec; [INSERT
AverageLatency(us)=532,02]
630 sec: 2973119 operations; 911,64 current ops/sec; [INSERT
AverageLatency(us)=47,38]
640 sec: 2973119 operations; 0 current ops/sec;
650 sec: 2973119 operations; 0 current ops/sec;
660 sec: 2973119 operations; 0 current ops/sec;
670 sec: 2973119 operations; 0 current ops/sec;
680 sec: 2973119 operations; 0 current ops/sec;
^[( 690 sec: 2977679 operations; 455,77 current ops/sec; [INSERT
AverageLatency(us)=14205,62]
700 sec: 2977679 operations; 0 current ops/sec;
710 sec: 2977679 operations; 0 current ops/sec;
720 sec: 3005039 operations; 2734,63 current ops/sec; [INSERT
AverageLatency(us)=1225,72]
730 sec: 3032399 operations; 2734,63 current ops/sec; [INSERT
AverageLatency(us)=185,11]
740 sec: 3032399 operations; 0 current ops/sec;
750 sec: 3032399 operations; 0 current ops/sec;
760 sec: 3032399 operations; 0 current ops/sec;
770 sec: 3032399 operations; 0 current ops/sec;
780 sec: 3032399 operations; 0 current ops/sec;
790 sec: 3064319 operations; 3190,72 current ops/sec; [INSERT
AverageLatency(us)=2069,81]
800 sec: 3260399 operations; 19598,2 current ops/sec; [INSERT
AverageLatency(us)=47,44]
810 sec: 3442799 operations; 18230,88 current ops/sec; [INSERT
AverageLatency(us)=50,83]
820 sec: 3620639 operations; 17776,89 current ops/sec; [INSERT
AverageLatency(us)=52,08]
830 sec: 3757439 operations; 13673,16 current ops/sec; [INSERT
AverageLatency(us)=58,18]
{noformat}
With 6295
{noformat}
10 sec: 71333 operations; 7126,17 current ops/sec; [INSERT
AverageLatency(us)=133,8]
20 sec: 159599 operations; 8823,07 current ops/sec; [INSERT
AverageLatency(us)=95,37]
30 sec: 212639 operations; 5301,35 current ops/sec; [INSERT
AverageLatency(us)=191,39]
40 sec: 284383 operations; 7171,53 current ops/sec; [INSERT
AverageLatency(us)=145,36]
50 sec: 348278 operations; 6386,31 current ops/sec; [INSERT
AverageLatency(us)=141,91]
60 sec: 453065 operations; 10473,46 current ops/sec; [INSERT
AverageLatency(us)=92,35]
70 sec: 552922 operations; 9981,71 current ops/sec; [INSERT
AverageLatency(us)=85,67]
80 sec: 654924 operations; 10195,1 current ops/sec; [INSERT
AverageLatency(us)=104,82]
90 sec: 779759 operations; 12478,51 current ops/sec; [INSERT
AverageLatency(us)=78,21]
100 sec: 779762 operations; 0,3 current ops/sec; [INSERT
AverageLatency(us)=343108,67]
110 sec: 913636 operations; 13380,71 current ops/sec; [INSERT
AverageLatency(us)=137,26]
120 sec: 1028664 operations; 11498,2 current ops/sec; [INSERT
AverageLatency(us)=85,91]
130 sec: 1141992 operations; 11310,18 current ops/sec; [INSERT
AverageLatency(us)=86,67]
140 sec: 1274264 operations; 13213,99 current ops/sec; [INSERT
AverageLatency(us)=70,65]
150 sec: 1412375 operations; 13783,53 current ops/sec; [INSERT
AverageLatency(us)=66,98]
160 sec: 1505215 operations; 9279,36 current ops/sec; [INSERT
AverageLatency(us)=103,44]
170 sec: 1630561 operations; 12529,59 current ops/sec; [INSERT
AverageLatency(us)=79,55]
180 sec: 1709829 operations; 7922,84 current ops/sec; [INSERT
AverageLatency(us)=122,32]
190 sec: 1739109 operations; 2926,54 current ops/sec; [INSERT
AverageLatency(us)=327,26]
200 sec: 1856100 operations; 11694,42 current ops/sec; [INSERT
AverageLatency(us)=76,57]
210 sec: 1965944 operations; 10978,91 current ops/sec; [INSERT
AverageLatency(us)=90,48]
220 sec: 2089603 operations; 12352,31 current ops/sec; [INSERT
AverageLatency(us)=82,49]
230 sec: 2160518 operations; 7090,08 current ops/sec; [INSERT
AverageLatency(us)=88,86]
240 sec: 2272941 operations; 11236,68 current ops/sec; [INSERT
AverageLatency(us)=115,35]
250 sec: 2367483 operations; 9450,42 current ops/sec; [INSERT
AverageLatency(us)=69,52]
260 sec: 2451594 operations; 8406,9 current ops/sec; [INSERT
AverageLatency(us)=148,02]
270 sec: 2554506 operations; 10286,06 current ops/sec; [INSERT
AverageLatency(us)=95,37]
280 sec: 2658804 operations; 10425,63 current ops/sec; [INSERT
AverageLatency(us)=93,62]
290 sec: 2711871 operations; 5304,05 current ops/sec; [INSERT
AverageLatency(us)=73,6]
300 sec: 2764352 operations; 5242,33 current ops/sec; [INSERT
AverageLatency(us)=271,32]
310 sec: 2776791 operations; 1243,15 current ops/sec; [INSERT
AverageLatency(us)=839,36]
320 sec: 2782455 operations; 566,12 current ops/sec; [INSERT
AverageLatency(us)=1764,07]
330 sec: 2907870 operations; 12530,22 current ops/sec; [INSERT
AverageLatency(us)=83,97]
340 sec: 3050598 operations; 14267,09 current ops/sec; [INSERT
AverageLatency(us)=60,58]
350 sec: 3153205 operations; 10255,57 current ops/sec; [INSERT
AverageLatency(us)=92,51]
360 sec: 3158971 operations; 576,37 current ops/sec; [INSERT
AverageLatency(us)=1880,05]
370 sec: 3186783 operations; 2779,81 current ops/sec; [INSERT
AverageLatency(us)=350,07]
380 sec: 3190037 operations; 325,14 current ops/sec; [INSERT
AverageLatency(us)=3147,28]
390 sec: 3282451 operations; 9240,48 current ops/sec; [INSERT
AverageLatency(us)=104,57]
400 sec: 3456366 operations; 17381,07 current ops/sec; [INSERT
AverageLatency(us)=53,3]
410 sec: 3576864 operations; 12046,19 current ops/sec; [INSERT
AverageLatency(us)=72,7]
420 sec: 3743489 operations; 16654,17 current ops/sec; [INSERT
AverageLatency(us)=60,94]
430 sec: 3929387 operations; 18580,51 current ops/sec; [INSERT
AverageLatency(us)=50,29]
440 sec: 4030892 operations; 10146,44 current ops/sec; [INSERT
AverageLatency(us)=85,66]
450 sec: 4167314 operations; 13635,38 current ops/sec; [INSERT
AverageLatency(us)=73,91]
460 sec: 4296216 operations; 12883,76 current ops/sec; [INSERT
AverageLatency(us)=75,24]
470 sec: 4431010 operations; 13474,01 current ops/sec; [INSERT
AverageLatency(us)=58,89]
480 sec: 4581058 operations; 14997,3 current ops/sec; [INSERT
AverageLatency(us)=75,31]
490 sec: 4657821 operations; 7672,46 current ops/sec; [INSERT
AverageLatency(us)=113,6]
500 sec: 4751360 operations; 9350,16 current ops/sec; [INSERT
AverageLatency(us)=109]
510 sec: 4760708 operations; 934,33 current ops/sec; [INSERT
AverageLatency(us)=1038,45]
520 sec: 4828627 operations; 6789,18 current ops/sec; [INSERT
AverageLatency(us)=139,98]
530 sec: 4948462 operations; 11977,51 current ops/sec; [INSERT
AverageLatency(us)=87,31]
540 sec: 5050637 operations; 10212,39 current ops/sec; [INSERT
AverageLatency(us)=91,17]
550 sec: 5121323 operations; 7060,13 current ops/sec; [INSERT
AverageLatency(us)=142,1]
560 sec: 5168196 operations; 4684,96 current ops/sec; [INSERT
AverageLatency(us)=207,26]
570 sec: 5253669 operations; 8227,26 current ops/sec; [INSERT
AverageLatency(us)=107,41]
580 sec: 5254516 operations; 84,67 current ops/sec; [INSERT
AverageLatency(us)=3862,3]
590 sec: 5408430 operations; 15374,49 current ops/sec; [INSERT
AverageLatency(us)=111,53]
600 sec: 5468717 operations; 6027,49 current ops/sec; [INSERT
AverageLatency(us)=121,72]
610 sec: 5488564 operations; 1983,71 current ops/sec; [INSERT
AverageLatency(us)=540]
620 sec: 5503674 operations; 1510,24 current ops/sec; [INSERT
AverageLatency(us)=586,2]
630 sec: 5580591 operations; 7688,62 current ops/sec; [INSERT
AverageLatency(us)=156,3]
640 sec: 5667381 operations; 8674,66 current ops/sec; [INSERT
AverageLatency(us)=114,85]
650 sec: 5824473 operations; 15701,35 current ops/sec; [INSERT
AverageLatency(us)=58,37]
660 sec: 5863498 operations; 3900,94 current ops/sec; [INSERT
AverageLatency(us)=186,93]
670 sec: 5867286 operations; 378,61 current ops/sec; [INSERT
AverageLatency(us)=1906,5]
680 sec: 5881480 operations; 1418,83 current ops/sec; [INSERT
AverageLatency(us)=969,62]
690 sec: 5918807 operations; 3730,83 current ops/sec; [INSERT
AverageLatency(us)=300,35]
700 sec: 5926122 operations; 731,13 current ops/sec; [INSERT
AverageLatency(us)=1207,91]
710 sec: 6115204 operations; 18900,64 current ops/sec; [INSERT
AverageLatency(us)=50,77]
720 sec: 6116016 operations; 80,26 current ops/sec; [INSERT
AverageLatency(us)=12618,78]
731 sec: 6168475 operations; 5043,65 current ops/sec; [INSERT
AverageLatency(us)=194,3]
741 sec: 6169493 operations; 99,1 current ops/sec; [INSERT
AverageLatency(us)=5123,27]
751 sec: 6206016 operations; 3518,25 current ops/sec; [INSERT
AverageLatency(us)=391,2]
761 sec: 6231541 operations; 2550,46 current ops/sec; [INSERT
AverageLatency(us)=467,12]
771 sec: 6252864 operations; 2131,23 current ops/sec; [INSERT
AverageLatency(us)=430,23]
781 sec: 6304020 operations; 5104,88 current ops/sec; [INSERT
AverageLatency(us)=202,38]
791 sec: 6328025 operations; 2399,54 current ops/sec; [INSERT
AverageLatency(us)=368,89]
801 sec: 6337032 operations; 900,25 current ops/sec; [INSERT
AverageLatency(us)=1175,08]
812 sec: 6345672 operations; 863,57 current ops/sec; [INSERT
AverageLatency(us)=1005,39]
822 sec: 6369258 operations; 2339,88 current ops/sec; [INSERT
AverageLatency(us)=409,83]
832 sec: 6380343 operations; 1107,95 current ops/sec; [INSERT
AverageLatency(us)=763,3]
842 sec: 6391880 operations; 1153,24 current ops/sec; [INSERT
AverageLatency(us)=1177,43]
852 sec: 6460288 operations; 6837,38 current ops/sec; [INSERT
AverageLatency(us)=143,37]
862 sec: 6465474 operations; 504,82 current ops/sec; [INSERT
AverageLatency(us)=1027,68]
872 sec: 6496268 operations; 3078,17 current ops/sec; [INSERT
AverageLatency(us)=296,25]
882 sec: 6531310 operations; 3502,45 current ops/sec; [INSERT
AverageLatency(us)=439,09]
892 sec: 6541217 operations; 990,2 current ops/sec; [INSERT
AverageLatency(us)=873,16]
{noformat}
So, in this test
- We nearly twice faster (for some tests the gap was not that huge, but still
over 80%).
- We don't have period of 20s or more where we do nothing (hence that's likely
why we're twice faster at the end).
Bad news beeing:
- it crashes a lot, whatever the version (plain trunk or trunk with 6295).
> Possible performance improvement in client batch operations: presplit and
> send in background
> --------------------------------------------------------------------------------------------
>
> Key: HBASE-6295
> URL: https://issues.apache.org/jira/browse/HBASE-6295
> Project: HBase
> Issue Type: Improvement
> Components: Client, Performance
> Affects Versions: 0.95.2
> Reporter: Nicolas Liochon
> Assignee: Nicolas Liochon
> Labels: noob
> Attachments: 6295.v1.patch, 6295.v2.patch, 6295.v3.patch,
> 6295.v4.patch
>
>
> today batch algo is:
> {noformat}
> for Operation o: List<Op>{
> add o to todolist
> if todolist > maxsize or o last in list
> split todolist per location
> send split lists to region servers
> clear todolist
> wait
> }
> {noformat}
> We could:
> - create immediately the final object instead of an intermediate array
> - split per location immediately
> - instead of sending when the list as a whole is full, send it when there is
> enough data for a single location
> It would be:
> {noformat}
> for Operation o: List<Op>{
> get location
> add o to todo location.todolist
> if (location.todolist > maxLocationSize)
> send location.todolist to region server
> clear location.todolist
> // don't wait, continue the loop
> }
> send remaining
> wait
> {noformat}
> It's not trivial to write if you add error management: retried list must be
> shared with the operations added in the todolist. But it's doable.
> It's interesting mainly for 'big' writes
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
