[
https://issues.apache.org/jira/browse/HBASE-17471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833930#comment-15833930
]
Yu Li commented on HBASE-17471:
-------------------------------
bq. MVCC and region seqid are totally different concept, combing then mixed up
wal append and memstore insertion.
Yes, it's true from the semantic view, but not strong enough to take the
rollback (re-separate) efforts IMO (and could you help introduce the historical
reason of separating them [~stack]? Thanks). From performance aspect
HBASE-14460 and HBASE-16698 are known issues, and please let us know If you
observed any more (real ones instead of conceptional) [~allan163]. If no more,
I think focusing on improving the mvcc preassign feature is enough. If quite
more, I think it worth a discussion on whether to re-separate mvcc and sequence
id.
bq. As I noticed in our tests, even with preAssign, some handler still need to
wait for the advance of mvcc numbers. Since we now begin a mvcc transition
before append, and complete them after sync. You can treat mvcc queue as a
lock, for now, this lock is holding by a single handler for a long time.
Actually this is exactly what I was concerned about the new design and the
reason asking for a carefully testing. I don't think we could remove all
locking stuff on mvcc, so I care more about the comparison data, say what's the
throughput of the implementation based on new design and the old one. Will wait
for more detailed data. Thanks.
> Region Seqid will be out of order in WAL if using mvccPreAssign
> ---------------------------------------------------------------
>
> Key: HBASE-17471
> URL: https://issues.apache.org/jira/browse/HBASE-17471
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 2.0.0, 1.4.0
> Reporter: Allan Yang
> Assignee: Allan Yang
> Priority: Critical
> Attachments: HBASE-17471-duo.patch, HBASE-17471-duo-v1.patch,
> HBASE-17471.patch, HBASE-17471.tmp, HBASE-17471.v2.patch, HBASE-17471.v3.patch
>
>
> mvccPreAssign was brought by HBASE-16698, which truly improved the
> performance of writing, especially in ASYNC_WAL scenario. But mvccPreAssign
> was only used in {{doMiniBatchMutate}}, not in Increment/Append path. If
> Increment/Append and batch put are using against the same region in parallel,
> then seqid of the same region may not monotonically increasing in the WAL.
> Since one write path acquires mvcc/seqid before append, and the other
> acquires in the append/sync consume thread.
> The out of order situation can easily reproduced by a simple UT, which was
> attached in the attachment. I modified the code to assert on the disorder:
> {code}
> if(this.highestSequenceIds.containsKey(encodedRegionName)) {
> assert highestSequenceIds.get(encodedRegionName) < sequenceid;
> }
> {code}
> I'd like to say, If we allow disorder in WALs, then this is not a issue.
> But as far as I know, if {{highestSequenceIds}} is not properly set, some
> WALs may not archive to oldWALs correctly.
> which I haven't figure out yet is that, will disorder in WAL cause data loss
> when recovering from disaster? If so, then it is a big problem need to be
> fixed.
> I have fix this problem in our costom1.1.x branch, my solution is using
> mvccPreAssign everywhere, making it un-configurable. Since mvccPreAssign it
> is indeed a better way than assign seqid in the ringbuffer thread while
> keeping handlers waiting for it.
> If anyone think it is doable, then I will port it to branch-1 and master
> branch and upload it.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
