[
https://issues.apache.org/jira/browse/HBASE-21856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16788236#comment-16788236
]
Andrew Purtell edited comment on HBASE-21856 at 3/8/19 7:41 PM:
----------------------------------------------------------------
There are two aspects of this:
1: Source side ordering. Sources need to ensure that all/any logs which contain
the edits for a given region and row are processed in order. Today we can have
the region open on one server generating edits and a recovery queue from a
crashed regionserver somewhere else, leading to interleaving. [~lhofhansl]'s
suggestions in the issue description are a solution for this. A strawman ideal
solution: Upon failure of a replication source, the replication queues are
split into per region queues, just like WALs are split into per region
recovered edits files. Each split-replication-queue is assigned to the current
replication source handling the region's edits. The source will drain the
recovery queue before shipping newer edits.
2. Sink side ordering. All of the work above is useless if sources then send
batches of work containing the edits for a given region and row to more than
one destination over at the sink. Who can say in what order the sinks will
process the replication RPC? They could be processed out or order or in
parallel. The sources should build replication batch RPCs by region (the code
today does this in part) and should chose a constant destination for a region's
batched edits with a consistent hash. (Multiple pending batches for the same
remote regionserver can be merged as an optimization.) Given a set of sink
endpoints and an encoded region name, always choose the same sink. This
guarantees at most one live endpoint in the sink will be processing a region's
edits, and so can apply the edits in a total order.
was (Author: apurtell):
There are two aspects of this:
1: Source side ordering. Sources need to ensure that all/any logs which contain
the edits for a given region and row are processed in order. Today we can have
the region open on one server generating edits and a recovery queue from a
crashed regionserver somewhere else, leading to interleaving. [~lhofhansl]'s
suggestions in the issue description are a solution for this. A strawman ideal
solution: Replication queues are split into per region queues, just like WALs
are split into per region recovered edits files. The new owner of the primary
region takes ownership of the split-replication-queue and drains the queue
before shipping newer edits.
2. Sink side ordering. All of the work above is useless if sources then send
batches of work containing the edits for a given region and row to more than
one destination over at the sink. Who can say in what order the sinks will
process the replication RPC? They could be processed out or order or in
parallel. The sources should build replication batch RPCs by region (the code
today does this in part) and should chose a constant destination for a region's
batched edits with a consistent hash. (Multiple pending batches for the same
remote regionserver can be merged as an optimization.) Given a set of sink
endpoints and an encoded region name, always choose the same sink. This
guarantees at most one live endpoint in the sink will be processing a region's
edits, and so can apply the edits in a total order.
> Consider Causal Replication Ordering
> ------------------------------------
>
> Key: HBASE-21856
> URL: https://issues.apache.org/jira/browse/HBASE-21856
> Project: HBase
> Issue Type: Brainstorming
> Reporter: Lars Hofhansl
> Priority: Major
>
> We've had various efforts to improve the ordering guarantees for HBase
> replication, most notably Serial Replication.
> I think in many cases guaranteeing a Total Replication Order is not required,
> but a simpler Causal Replication Order is sufficient.
> Specifically we would guarantee causal ordering for a single Rowkey. Any
> changes to a Row - Puts, Deletes, etc - would be replicated in the exact
> order in which they occurred in the source system.
> Unlike total ordering this can be accomplished with only local region server
> control.
> I don't have a full design in mind, let's discuss here. It should be
> sufficient to to the following:
> # RegionServers only adopt the replication queues from other RegionServers
> for regions they (now) own. This requires log splitting for replication.
> # RegionServers ship all edits for queues adopted from other servers before
> any of their "own" edits are shipped.
> It's probably a bit more involved, but should be much cheaper that the total
> ordering provided by serial replication.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
