[
https://issues.apache.org/jira/browse/HBASE-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709980#action_12709980
]
Jim Kellerman commented on HBASE-1177:
--------------------------------------
When the application runs on the same host as the one serving the table, there
is a lot more context switching going on which accounts for the differences in
times. See below:
{code}
Application running on a host different from the one serving the table:
Read 1 row with 7 columns 100 times in 108ms
Read 1 row with 8 columns 100 times in 111ms
Read 1 row with 1000 columns 100 times in 8995ms
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 1 112 182440 76356 6429056 0 0 282 1955 7 6 4 3 86 6 0
0 0 112 182532 76368 6429112 0 0 16 40 25 335 0 0 100 0
0
0 0 112 182532 76368 6429112 0 0 0 0 15 312 0 0 100 0
0
0 0 112 182656 76368 6429112 0 0 0 0 16 330 0 0 100 0
0
0 0 112 182896 76368 6429112 0 0 0 0 16 305 0 0 100 0
0
0 0 112 182896 76376 6429112 0 0 0 40 28 348 0 0 100 0
0
0 0 112 182896 76380 6429112 0 0 0 36 28 341 0 0 100 0
0
1 0 112 177516 76380 6429144 0 0 0 0 17 511 1 1 98 0 0
0 0 112 153648 76380 6429144 0 0 0 0 128 1164 33 2 65 0 0
0 0 112 153396 76380 6429152 0 0 0 5 45 694 1 0 99 0 0
0 0 112 153396 76380 6429152 0 0 0 44 18 342 0 0 100 0
0
2 0 112 153388 76388 6429152 0 0 0 16 29 480 0 0 100 0
0
0 0 112 153388 76392 6429152 0 0 0 64 27 380 0 0 100 0
0
0 0 112 153132 76392 6429152 0 0 0 1 29 487 1 0 99 0 0
0 0 112 153008 76392 6429152 0 0 0 0 10 492 0 0 100 0
0
0 0 112 141904 76392 6429992 0 0 0 25 1901 4647 21 2 77 0 0
0 0 112 135344 76392 6429992 0 0 0 0 1693 528 4 2 94 0 0
0 0 112 131084 76404 6429992 0 0 0 48 1886 411 1 3 96 0 0
0 0 112 119920 76404 6429992 0 0 0 0 1769 2398 4 1 95 0 0
0 0 112 117428 76408 6429992 0 0 0 24 1972 461 2 1 97 0 0
0 0 112 117300 76408 6429992 0 0 0 12 1810 386 1 1 97 0 0
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 112 108104 76408 6429992 0 0 0 0 1944 445 6 1 92 0 0
0 0 112 83452 76408 6429992 0 0 0 0 1746 456 16 2 82 0 0
0 0 112 83796 76408 6429992 0 0 0 0 1492 448 0 2 98 0 0
0 0 112 83608 76420 6429992 0 0 0 57 897 411 0 1 99 0 0
0 0 112 180416 76420 6429960 0 0 0 0 30 414 0 0 100 0
0
0 0 112 180340 76420 6429960 0 0 0 0 20 326 0 0 100 0
0
0 0 112 180340 76420 6429960 0 0 0 0 15 311 0 0 100 0
0
0 0 112 180676 76420 6429960 0 0 0 0 15 328 0 0 100 0
0
0 0 112 180680 76420 6429960 0 0 0 0 14 321 0 0 100 0
0
0 0 112 180680 76420 6429960 0 0 0 77 35 350 0 0 100 0
0
0 0 112 180804 76420 6429960 0 0 0 0 23 323 0 0 100 0
0
0 0 112 180804 76420 6429960 0 0 0 0 10 308 0 0 100 0
0
0 0 112 180928 76428 6429960 0 0 24 16 33 380 4 0 95 1 0
0 0 112 180556 76428 6430036 0 0 52 0 46 458 4 0 95 1 0
0 0 112 178656 76436 6430036 0 0 0 112 40 1288 1 0 99 0 0
0 0 112 178820 76436 6430036 0 0 0 0 20 325 0 0 100 0
0
2 0 112 178820 76436 6430036 0 0 0 0 24 352 0 0 100 0
0
1 0 112 178824 76436 6430036 0 0 0 0 13 337 0 0 100 0
0
0 0 112 179136 76436 6430036 0 0 0 0 22 414 0 0 100 0
0
0 0 112 179136 76436 6430036 0 0 0 40 18 354 0 0 100 0
0
0 0 112 178840 76440 6430036 0 0 0 28 44 420 0 0 100 0
0
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 112 178880 76448 6430036 0 0 0 16 31 342 0 0 100 0
0
0 0 112 178288 76448 6430036 0 0 0 0 15 525 1 1 98 0 0
0 0 112 178384 76448 6430040 0 0 0 0 13 322 0 0 100 0
0
0 0 112 178384 76448 6430040 0 0 0 919 36 335 0 0 100 0
0
1 0 112 178384 76452 6430040 0 0 0 48 26 357 0 0 100 0
0
0 0 112 178536 76452 6430040 0 0 0 0 41 327 0 0 100 0
0
0 0 112 178544 76452 6430040 0 0 0 0 42 330 0 0 100 0
0
1 0 112 178560 76452 6430040 0 0 0 0 47 328 0 0 100 0
0
0 0 112 178684 76452 6430040 0 0 0 30 37 347 0 0 100 0
0
0 0 112 178676 76460 6430040 0 0 0 60 41 399 0 0 100 0
0
0 0 112 178684 76460 6430040 0 0 0 0 19 336 0 0 100 0
0
0 0 112 178684 76460 6430040 0 0 0 0 23 337 0 0 100 0
0
1 0 112 178808 76460 6430040 0 0 0 0 16 331 0 0 100 0
0
1 0 112 178932 76460 6430040 0 0 0 8 14 338 0 0 100 0
0
0 0 112 178932 76460 6430040 0 0 0 0 30 326 0 0 100 0
0
0 0 112 178784 76464 6430040 0 0 0 16 12 403 0 0 100 0
0
0 0 112 178808 76464 6430040 0 0 0 0 13 330 0 0 100 0
0
0 0 112 178808 76464 6430040 0 0 0 0 6 338 0 0 100 0
0
0 0 112 178932 76472 6430040 0 0 0 25 20 357 0 0 100 0
0
1 0 112 179056 76472 6430040 0 0 0 0 17 323 0 0 100 0
0
1 0 112 179056 76472 6430040 0 0 0 0 8 316 0 0 100 0
0
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 112 179172 76480 6430040 0 0 0 20 20 436 0 0 100 0
0
0 0 112 179296 76480 6430040 0 0 0 0 24 319 0 0 100 0
0
0 0 112 179296 76480 6430040 0 0 0 28 51 329 0 0 100 0
0
0 0 112 179296 76480 6430040 0 0 0 0 33 335 0 0 100 0
0
1 0 112 179384 76480 6430040 0 0 0 0 13 338 0 0 100 0
0
0 0 112 179508 76480 6430040 0 0 0 0 10 310 0 0 100 0
0
0 0 112 179508 76492 6430044 0 0 0 65 31 373 0 0 100 0
0
0 0 112 179508 76492 6430044 0 0 0 16 39 334 0 0 100 0
0
Application running on the same host as the one serving the table:
Read 1 row with 7 columns 100 times in 97ms
Read 1 row with 8 columns 100 times in 3997ms
Read 1 row with 1000 columns 100 times in 10460ms
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
2 0 116 35120 89092 6380936 0 0 307 1993 5 13 5 4 85 6 0
0 0 116 34988 89092 6380936 0 0 0 4 29 385 0 0 100 0
0
1 0 116 34988 89092 6380936 0 0 0 0 3 320 0 0 100 0
0
1 0 116 34492 89096 6380936 0 0 0 36 22 425 0 0 100 0
0
1 0 116 34616 89096 6380936 0 0 0 0 27 354 0 0 100 0
0
1 0 116 34616 89096 6380936 0 0 0 0 13 338 0 0 100 0
0
1 0 116 34864 89104 6380936 0 0 0 20 21 361 0 0 100 0
0
0 0 116 34912 89104 6380936 0 0 0 0 12 351 0 0 100 0
0
1 0 116 34912 89104 6380936 0 0 0 28 16 337 0 0 100 0
0
1 0 116 28964 89104 6371916 0 0 0 0 20 964 12 2 86 0 0
0 0 116 28388 89104 6358320 0 0 0 0 183 1078 22 1 77 0 0
2 0 116 28384 89104 6358320 0 0 0 69 58 358 0 0 100 0
0
0 0 116 27880 89112 6358312 0 0 0 4 111 937 0 1 99 0 0
0 0 116 27832 89112 6358316 0 0 0 0 26 396 0 0 100 0
0
1 0 116 27840 89116 6358316 0 0 0 96 41 386 0 0 100 0
0
0 0 116 27964 89116 6358316 0 0 0 0 23 404 0 0 100 0
0
4 0 116 25128 89116 6353476 0 0 0 58 134 774 14 0 85 0 0
1 0 116 27368 89124 6351088 0 0 0 32 258 1977 4 1 96 0 0
0 0 116 27368 89124 6351096 0 0 0 8 71 674 0 0 100 0
0
0 0 116 26684 89124 6351096 0 0 0 6 13 710 0 0 100 0
0
3 0 116 26960 89132 6351096 0 0 0 72 26 645 0 0 100 0
0
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
3 0 116 28552 89132 6342908 0 0 0 26 37 688 24 1 74 0 0
1 0 116 26540 89132 6326496 0 0 0 0 21 1201 22 2 76 0 0
1 0 116 27480 89140 6318244 0 0 0 16 19 1450 22 1 76 0 0
1 0 116 28620 89140 6314104 0 0 0 0 26 2092 21 1 78 0 0
0 0 116 40952 89148 6314096 0 0 0 28 40 3456 21 1 77 0 0
3 0 116 40080 89148 6314104 0 0 0 12 25 1406 25 0 75 0 0
1 0 116 33888 89148 6314104 0 0 0 0 20 1480 27 0 73 0 0
1 0 116 24888 89148 6309924 0 0 0 0 17 1564 17 1 82 0 0
2 0 116 27952 89156 6305776 0 0 0 16 20 1594 19 0 81 0 0
0 0 116 28156 89156 6305788 0 0 0 0 24 1499 18 0 82 0 0
0 0 116 42532 89164 6305788 0 0 0 32 40 990 9 0 91 0 0
0 0 116 122276 89164 6305788 0 0 0 0 17 438 0 0 100 0
0
0 0 116 122256 89164 6305756 0 0 0 0 19 425 0 0 100 0
0
0 0 116 122552 89164 6305756 0 0 0 0 21 362 0 0 100 0
0
0 0 116 122676 89172 6305748 0 0 0 32 31 361 0 0 100 0
0
0 0 116 122676 89172 6305756 0 0 0 40 35 365 0 0 100 0
0
0 0 116 123048 89172 6305756 0 0 0 0 67 367 0 0 100 0
0
0 0 116 123048 89172 6305756 0 0 0 0 43 336 0 0 100 0
0
0 0 116 123048 89172 6305756 0 0 0 0 57 380 0 0 100 0
0
0 0 116 123544 89172 6305756 0 0 0 0 27 354 0 0 100 0
0
0 0 116 123544 89180 6305748 0 0 0 28 16 372 0 0 100 0
0
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 116 123544 89184 6305756 0 0 0 24 18 361 0 0 100 0
0
0 0 116 123820 89184 6305756 0 0 0 0 20 356 0 0 100 0
0
0 0 116 123820 89184 6305760 0 0 0 0 17 355 0 0 100 0
0
0 0 116 123944 89184 6305760 0 0 0 0 23 388 0 0 100 0
0
0 0 116 124184 89184 6305760 0 0 0 56 23 353 0 0 100 0
0
0 0 116 124184 89200 6305744 0 0 0 89 39 392 0 0 100 0
0
0 0 116 124184 89200 6305760 0 0 0 0 16 376 0 0 100 0
0
0 0 116 124308 89200 6305760 0 0 0 0 12 366 0 0 100 0
0
{code}
I think this issue is a red herring, and there is no basic difference between
running the test program on the same host as the one serving the table vs
running the test program on a different server.
The difference when running on the same server is that the OS needs to do
context switches between the client application and the region server.
> Delay when client is located on the same node as the regionserver
> -----------------------------------------------------------------
>
> Key: HBASE-1177
> URL: https://issues.apache.org/jira/browse/HBASE-1177
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: Linux 2.6.25 x86_64
> Reporter: Jonathan Gray
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: ReadDelayTest.java
>
>
> During testing of HBASE-80, we uncovered a strange 40ms delay for random
> reads. We ran a series of tests and found that it only happens when the
> client is on the same node as the RS and for a certain range of payloads (not
> specifically related to number of columns or size of them, only total
> payload). It appears to be precisely 40ms every time.
> Unsure if this is particular to our architecture, but it does happen on all
> nodes we've tried. Issue completely goes away with very large payloads or
> moving the client.
> Will post a test program tomorrow if anyone can test on a different
> architecture.
> Making a blocker for 0.20. Since this happens when you have an MR task
> running local to the RS, and this is what we try to do, might also consider
> making this a blocker for 0.19.1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
