[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15135390#comment-15135390 ] stack commented on HBASE-14460: --- Quick update. HBASE-15213 fixes mvcc. Means we get our perf back for increment, append, and checkAnd* on branch-1.0 and branch-1.1. The same issue, HBASE-15213, identifies that hbase-1.2.0 does not suffer the performance regression. It did, but it got fixed by HBASE-12751 which does similar to HBASE-15213. So, the work done to make an increment 'fast path' in HBASE-15031 for branch-1.0+ where we skirt mvcc when incrementing is now no longer needed. Let me actually remove this option (after HBASE-15213 goes in). Will be back with cleaner messaging on release notes and up on dev mailing list after all gets committed. > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132944#comment-15132944 ] stack commented on HBASE-14460: --- Excellent analysis of why the friction in mvcc in branch-1.0, branch-1.1. > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112690#comment-15112690 ] stack commented on HBASE-14460: --- Hey [~bbeaudreault] I made HBASE-15159 a subtask here. I don't think it'd be too hard making the branch-1 hack for increments work for checkAnd*... let me take a look. Doing it on a per table or per column-family basis might be tough. Let me report back on HBASE-15159. > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15112673#comment-15112673 ] Bryan Beaudreault commented on HBASE-14460: --- Thanks for the work here guys. A couple questions/comments: - We are also running into this issue pretty regularly for checkAndPut. A burst of checkAndPuts can easily bring a RegionServer to it's knees. If we could apply the same fix made for Increments to checkAnd* operations, that would be great. - The vast majority of our Increment and checkAnd* usage could safely use this, but given our environment it's hard to make that statement cluster-wide. It would be *very* nice if this config could be enable for just particular tables (or, if possible, CFs within a table). > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15111221#comment-15111221 ] stack commented on HBASE-14460: --- Added release note on approach going on in here. (Unfortunately,) We have three different changes made under the rubric of this issue. In the branch-1.0 (and branch-1.1) fix, HBASE-15031, and in the branch-1.2+ fix, HBASE-15091, we address the Increment slow down only; you set a flag to take a 'fast path' for Increments only. The patches for branch-1.0+branch-1.1 and that for branch-1.2+ differ pretty substantially mostly because HBASE-12751 changed the theatre of operation. The fix for master branch is a general fix. It builds on the changes in branch-1 taking the 'fast path' only for increments but then changes the write path for all updates in hbase -- appends, checkAnd* AND the ho-hum batchMutate, doMiniBatchMutate, etc., -- to do similar, making all changes under row lock, including sync and mvcc catchup, so append and checkAnd* can take the same 'fast path' and we can continue to honor our consistency-on-a-row tenet. Doing all under a lock was intentionally avoided in the past for performance reasons but HBASE-12751 (row locks are read/write rather than xclusive) plus all that has changed in the write path since changes the lay of the land; HBASE-15046 seems to show the default path runs a little faster though all now done under lock (more perf testing to follow). > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
