[
https://issues.apache.org/jira/browse/HBASE-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717369#action_12717369
]
Jim Kellerman commented on HBASE-1177:
--------------------------------------
This problem still exists in trunk after HBASE-1304 was committed. I'm going to
instrument the RPC layer tomorrow.
{code}
Read 1 row with 6 columns 100 times in 53 ms
Read 1 row with 7 columns 100 times in 81 ms
Read 1 row with 8 columns 100 times in 4133 ms
Read 1 row with 9 columns 100 times in 4000 ms
Read 1 row with 10 columns 100 times in 4000 ms
Read 1 row with 11 columns 100 times in 4000 ms
Read 1 row with 12 columns 100 times in 4000 ms
Read 1 row with 13 columns 100 times in 4000 ms
Read 1 row with 14 columns 100 times in 4000 ms
Read 1 row with 15 columns 100 times in 4000 ms
Read 1 row with 16 columns 100 times in 4010 ms
Read 1 row with 17 columns 100 times in 4000 ms
Read 1 row with 18 columns 100 times in 4000 ms
Read 1 row with 19 columns 100 times in 4000 ms
Read 1 row with 20 columns 100 times in 4000 ms
Read 1 row with 21 columns 100 times in 4000 ms
Read 1 row with 22 columns 100 times in 4000 ms
Read 1 row with 23 columns 100 times in 4000 ms
Read 1 row with 24 columns 100 times in 83 ms
Read 1 row with 25 columns 100 times in 72 ms
Read 1 row with 26 columns 100 times in 63 ms
Read 1 row with 27 columns 100 times in 66 ms
Read 1 row with 28 columns 100 times in 71 ms
Read 1 row with 29 columns 100 times in 67 ms
Read 1 row with 30 columns 100 times in 88 ms
Read 1 row with 31 columns 100 times in 78 ms
Read 1 row with 32 columns 100 times in 174 ms
Read 1 row with 33 columns 100 times in 208 ms
Read 1 row with 34 columns 100 times in 62 ms
Read 1 row with 35 columns 100 times in 130 ms
Read 1 row with 36 columns 100 times in 137 ms
Read 1 row with 37 columns 100 times in 142 ms
Read 1 row with 38 columns 100 times in 279 ms
Read 1 row with 39 columns 100 times in 184 ms
Read 1 row with 40 columns 100 times in 367 ms
Read 1 row with 41 columns 100 times in 202 ms
Read 1 row with 42 columns 100 times in 452 ms
Read 1 row with 43 columns 100 times in 206 ms
Read 1 row with 44 columns 100 times in 669 ms
Read 1 row with 45 columns 100 times in 242 ms
Read 1 row with 46 columns 100 times in 199 ms
Read 1 row with 47 columns 100 times in 250 ms
Read 1 row with 48 columns 100 times in 62 ms
Read 1 row with 49 columns 100 times in 62 ms
Read 1 row with 50 columns 100 times in 81 ms
Read 1 row with 51 columns 100 times in 66 ms
Read 1 row with 52 columns 100 times in 65 ms
Read 1 row with 53 columns 100 times in 64 ms
Read 1 row with 54 columns 100 times in 65 ms
Read 1 row with 55 columns 100 times in 113 ms
Read 1 row with 56 columns 100 times in 89 ms
Read 1 row with 57 columns 100 times in 72 ms
Read 1 row with 58 columns 100 times in 67 ms
Read 1 row with 59 columns 100 times in 67 ms
Read 1 row with 60 columns 100 times in 66 ms
Read 1 row with 61 columns 100 times in 78 ms
Read 1 row with 62 columns 100 times in 88 ms
Read 1 row with 63 columns 100 times in 72 ms
Read 1 row with 64 columns 100 times in 85 ms
Read 1 row with 65 columns 100 times in 82 ms
Read 1 row with 66 columns 100 times in 122 ms
Read 1 row with 67 columns 100 times in 82 ms
Read 1 row with 68 columns 100 times in 110 ms
Read 1 row with 69 columns 100 times in 168 ms
Read 1 row with 70 columns 100 times in 85 ms
Read 1 row with 71 columns 100 times in 85 ms
Read 1 row with 72 columns 100 times in 86 ms
Read 1 row with 73 columns 100 times in 109 ms
Read 1 row with 74 columns 100 times in 85 ms
Read 1 row with 75 columns 100 times in 97 ms
Read 1 row with 76 columns 100 times in 108 ms
Read 1 row with 77 columns 100 times in 128 ms
Read 1 row with 78 columns 100 times in 91 ms
Read 1 row with 79 columns 100 times in 102 ms
Read 1 row with 80 columns 100 times in 221 ms
Read 1 row with 81 columns 100 times in 138 ms
Read 1 row with 82 columns 100 times in 260 ms
Read 1 row with 83 columns 100 times in 474 ms
Read 1 row with 84 columns 100 times in 99 ms
Read 1 row with 85 columns 100 times in 110 ms
Read 1 row with 86 columns 100 times in 93 ms
Read 1 row with 87 columns 100 times in 313 ms
Read 1 row with 88 columns 100 times in 231 ms
Read 1 row with 89 columns 100 times in 246 ms
Read 1 row with 90 columns 100 times in 189 ms
Read 1 row with 91 columns 100 times in 162 ms
Read 1 row with 92 columns 100 times in 153 ms
Read 1 row with 93 columns 100 times in 157 ms
Read 1 row with 94 columns 100 times in 144 ms
Read 1 row with 95 columns 100 times in 156 ms
Read 1 row with 96 columns 100 times in 172 ms
Read 1 row with 97 columns 100 times in 362 ms
Read 1 row with 98 columns 100 times in 139 ms
Read 1 row with 99 columns 100 times in 171 ms
Read 1 row with 100 columns 100 times in 103 ms
Read 1 row with 101 columns 100 times in 149 ms
Read 1 row with 102 columns 100 times in 104 ms
Read 1 row with 103 columns 100 times in 410 ms
Read 1 row with 104 columns 100 times in 594 ms
Read 1 row with 1000 columns 100 times in 2041 ms
{code}
> Delay when client is located on the same node as the regionserver
> -----------------------------------------------------------------
>
> Key: HBASE-1177
> URL: https://issues.apache.org/jira/browse/HBASE-1177
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Environment: Linux 2.6.25 x86_64
> Reporter: Jonathan Gray
> Assignee: Jim Kellerman
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: Contribution of getClosest to getRow time.jpg,
> Contribution of next to getRow time.jpg, Contribution of seekTo to getClosest
> time.jpg, Elapsed time of RowResults.readFields.jpg, getRow + round-trip vs #
> columns.jpg, getRow times.jpg, ReadDelayTest.java, RowResults.readFields
> zoomed.jpg, screenshot-1.jpg, screenshot-2.jpg, screenshot-3.jpg,
> screenshot-4.jpg, zoom of columns vs round-trip blowup.jpg
>
>
> During testing of HBASE-80, we uncovered a strange 40ms delay for random
> reads. We ran a series of tests and found that it only happens when the
> client is on the same node as the RS and for a certain range of payloads (not
> specifically related to number of columns or size of them, only total
> payload). It appears to be precisely 40ms every time.
> Unsure if this is particular to our architecture, but it does happen on all
> nodes we've tried. Issue completely goes away with very large payloads or
> moving the client.
> Will post a test program tomorrow if anyone can test on a different
> architecture.
> Making a blocker for 0.20. Since this happens when you have an MR task
> running local to the RS, and this is what we try to do, might also consider
> making this a blocker for 0.19.1.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
