[
https://issues.apache.org/jira/browse/HBASE-15213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132543#comment-15132543
]
Junegunn Choi commented on HBASE-15213:
---------------------------------------
More data:
Similar improvement in Appends.
|| Num threads || branch-1.0 || branch-1.0 with fix ||
|| 1 | 2213 | 2451 |
|| 2 | 4849 | 5071 |
|| 4 | 7580 | 7723 |
|| 8 | 11287 | 11253 |
|| 16 | 8906 | 14945 |
|| 32 | 5332 | 17905 |
|| 64 | 3953 | 19394 |
|| 96 | 3780 | 19417 |
About master:
The patch I attached cannot be applied to master branch due to the major change
made on MultiVersionConcurrencyControl in HBASE-12751 which effectively
reverted the change from HBASE-8763 we are discussing here. I got curious and
measured the increment performance before and after HBASE-12751 and found out
that it almost recovered the performance.
|| Num threads || Before HBASE-12751 || After HBASE-12751 ||
|| 1 | 1956 | 2342 |
|| 2 | 3892 | 4168 |
|| 4 | 6624 | 7155 |
|| 8 | 8869 | 10587 |
|| 16 | 9045 | 13586 |
|| 32 | 4956 | 16299 |
|| 64 | 3767 | 18484 |
|| 96 | 3599 | 18993 |
The numbers of the recent master are not much different. So there's no reason
to apply this patch to master (because it's already done, effectively), but
considering that HBASE-12751 is not coming to HBase 1.0 ~ 1.2, and this patch
is pretty small, I think we can still consider having this on branch-1.0 ~ 1.2.
> Fix increment performance regression caused by HBASE-8763 on branch-1.0
> -----------------------------------------------------------------------
>
> Key: HBASE-15213
> URL: https://issues.apache.org/jira/browse/HBASE-15213
> Project: HBase
> Issue Type: Bug
> Components: Performance
> Reporter: Junegunn Choi
> Assignee: Junegunn Choi
> Attachments: HBASE-15213.branch-1.0.patch
>
>
> This is an attempt to fix the increment performance regression caused by
> HBASE-8763 on branch-1.0.
> I'm aware that hbase.increment.fast.but.narrow.consistency was added to
> branch-1.0 (HBASE-15031) to address the issue and a separate work is ongoing
> on master branch, but anyway, this is my take on the problem.
> I read through HBASE-14460 and HBASE-8763 but it wasn't clear to me what
> caused the slowdown but I could indeed reproduce the performance regression.
> Test setup:
> - Server: 4-core Xeon 2.4GHz Linux server running mini cluster (100 handlers,
> JDK 1.7)
> - Client: Another box of the same spec
> - Increments on random 10k records on a single-region table, recreated every
> time
> Increment throughput (TPS):
> || Num threads || Before HBASE-8763 (d6cc2fb) || branch-1.0 || branch-1.0
> (narrow-consistency) ||
> || 1 | 2661 | 2486 | 2359 |
> || 2 | 5048 | 5064 | 4867 |
> || 4 | 7503 | 8071 | 8690 |
> || 8 | 10471 | 10886 | 13980 |
> || 16 | 15515 | 9418 | 18601 |
> || 32 | 17699 | 5421 | 20540 |
> || 64 | 20601 | 4038 | 25591 |
> || 96 | 19177 | 3891 | 26017 |
> We can clearly observe that the throughtput degrades as we increase the
> number of concurrent requests, which led me to believe that there's severe
> context switching overhead and I could indirectly confirm that suspicion with
> cs entry in vmstat output. branch-1.0 shows a much higher number of context
> switches even with much lower throughput.
> Here are the observations:
> - WriteEntry in the writeQueue can only be removed by the very handler that
> put it, only when it is at the front of the queue and marked complete.
> - Since a WriteEntry is marked complete after the wait-loop, only one entry
> can be removed at a time.
> - This stringent condition causes O(N^2) context switches where n is the
> number of concurrent handlers processing requests.
> So what I tried here is to mark WriteEntry complete before we go into
> wait-loop. With the change, multiple WriteEntries can be shifted at a time
> without context switches. I changed writeQueue to LinkedHashSet since fast
> containment check is needed as WriteEntry can be removed by any handler.
> The numbers look good, it's virtually identical to pre-HBASE-8763 era.
> || Num threads || branch-1.0 with fix ||
> || 1 | 2459 |
> || 2 | 4976 |
> || 4 | 8033 |
> || 8 | 12292 |
> || 16 | 15234 |
> || 32 | 16601 |
> || 64 | 19994 |
> || 96 | 20052 |
> So what do you think about it? Please let me know if I'm missing anything.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
