[
https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14460:
--------------------------
Attachment: client.test.patch
98.80.flamegraph-11428.svg
1.0.80.flamegraph-7932.svg
Attempts at reproducing the slowdown in the small have failed to pay off. I see
the roughly 2x difference but not the 7x claimed by the original poster.
With some help from Preston Koprivica, you need to have some friction in place
to see the issue; the friction gets amplified by mvcc wait.
Here is a test that clearly shows the problem. 0.98 is about 33% slower than
0.94 (0.98 added in mvcc) and then 1.0+ is about 10x the latency WHEN you have
80 clients running external to the regionserver process banging on it.
The flame graphs show us spending loads of time in mvcc waiting. The stack
trace is the SAME as for the tests in the small but we just seem to be waiting
overall longer. There is an amplification going on.
Looking at options:
[[email protected]]'s suggestion is a nice one. Will narrow what we have
to wait on. I tried disabling completely our wait-on-mvcc before we read at all
and this helps; we are only 3x slower than 0.98 (and 4x slower than 0.94).
Need some other bit of trickery to take us closer to what was there before.
> [Perf Regression] Merge of MVCC and SequenceId (HBASE-HBASE-8763) slowed
> Increments, CheckAndPuts, batch operations
> -------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-14460
> URL: https://issues.apache.org/jira/browse/HBASE-14460
> Project: HBase
> Issue Type: Bug
> Components: Performance
> Reporter: stack
> Assignee: stack
> Priority: Critical
> Attachments: 0.94.test.patch, 0.98.test.patch,
> 1.0.80.flamegraph-7932.svg, 14460.txt, 98.80.flamegraph-11428.svg,
> HBASE-14460-discussion.patch, client.test.patch,
> flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg,
> flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg,
> flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg,
> m.test.patch, region_lock.png, testincrement.094.patch,
> testincrement.098.patch, testincrement.master.patch
>
>
> As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation
> between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of
> sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to
> 'catch up' to our current point before we can read the last Increment value
> that we need to update.
> We can say that our Increment is just done wrong, we should just be writing
> Increments and summing on read, but checkAndPut as well as batching
> operations have the same issue. Fix.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
