[
https://issues.apache.org/jira/browse/HBASE-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Daniel Cryans updated HBASE-2129:
--------------------------------------
Attachment: HBASE-2129.patch
This first patch adds a new contrib called "mdc_replication". It is
master/slave and only supports 1 slave. The modifications done to the core
project are only there to enable the sub-classing and extended functionality
(like ZKW which has more friendly methods).
The package.html file gives an overview of the features and describes how to
deploy it on 2 clusters. Every user table is replicated, there's no scoping. I
also uses Hadoop RPC as it was already integrated, in the future the plan is to
use a non-versioned RPC mechanism.
Replication works by first collecting the HLog.Entry in a queue on the master
cluster and are sent in a batch every 10 seconds by default. On the slave
cluster those edits are stored in a log file and then replayed by a separate
thread. It is currently blocking so while the edits are played no other batch
of edits can come in.
A region server decides on which RS to replicate to by scanning the /rs
directory in the slaves ZK folder and then randomly chooses a subset of nodes
according to a default ration of 10%. This is to make sure that we can handle
clusters of different sizes. When the RS actually replicates a batch of edits,
it first chooses a random node from the subset. This is a cheap way to do try
to even the load on the slave cluster.
If a region server dies on the slave side, the RS on the master gets a new
subset and retries with a (hopefully) new node.
If a region server dies on the master side, the RS getting a region from it
will first replicate every edit found in the oldlogfile.
Apart from the new unit tests, I tested this patch on a single machine by
putting two fully distributed clusters of 1 region server that used the same
HDFS and ZK quorum (the machine is a i7, 12GB RAM, 2x500GB in RAID1). I was
able to run PE sequentialWrite 1 on the first cluster and read all the edits on
the slave cluster.
There is a big potential for refactoring (ReplicationSink works a bit like a
HLog) in this patch and some parts are done with something bigger in mind
(multiple slaves for example). So, at this point, comments on the general
layout are very welcomed but nitpicks won't be very useful since it this code
is meant to change a lot.
> Simple Master/Slave replication
> -------------------------------
>
> Key: HBASE-2129
> URL: https://issues.apache.org/jira/browse/HBASE-2129
> Project: Hadoop HBase
> Issue Type: Sub-task
> Reporter: Jean-Daniel Cryans
> Assignee: Jean-Daniel Cryans
> Fix For: 0.21.0
>
> Attachments: HBASE-2129.patch
>
>
> We should first build a very simple replication mechanism to validate our
> assumptions and get a feel of what replication is in this very distributed
> context.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
