[
https://issues.apache.org/jira/browse/HBASE-15867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17548578#comment-17548578
]
Duo Zhang commented on HBASE-15867:
-----------------------------------
The progress for the POC is good.
https://github.com/apache9/hbase/tree/HBASE-15867
Of course there are still lots of compile errors, especially in the test code.
There are some changes in the concepts mentioned above. This is the new
ReplicationQueueStorage interface:
https://github.com/Apache9/hbase/blob/HBASE-15867/hbase-replication/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueueStorage.java
This is the most important part.
{code}
/**
* Store the replication offset for the specific group of a replication queue.
* <p/>
* If the current replication queue does not exist yet, will create it
automatically.
* @param queueId the id of the replication queue
* @param walGroup the wal group
* @param offset the offset for a group
* @param lastSeqIds map with {encodedRegionName, sequenceId} pairs for
serial replication
*/
void setOffset(ReplicationQueueId queueId, String walGroup,
ReplicationGroupOffset offset,
Map<String, Long> lastSeqIds) throws ReplicationException;
/**
* Get the replication offset for all the groups of a replication queue.
* <p/>
* Usually used when setup a recovered replication queue.
* @param queueId the id of the replication queue
* @return the offset for all the groups of the given replication queue
*/
Map<String, ReplicationGroupOffset> getOffsets(ReplicationQueueId queueId)
throws ReplicationException;
/**
* Get a list of all queues for the specified region server.
* @param serverName the server name of the region server that owns the set
of queues
* @return a list of queueIds
*/
List<ReplicationQueueId> listAllQueueIds(ServerName serverName) throws
ReplicationException;
/**
* Get a list of all region servers that have outstanding replication queues.
These servers could
* be alive, dead or from a previous run of the cluster.
* @return a list of server names
*/
List<ServerName> listAllReplicators() throws ReplicationException;
/**
* Change ownership for the queue identified by queueId and belongs to a dead
region server.
* @param peerId the id of the replication peer
* @param queueId the id of the replication queue
* @param targetServerName the name of the target region server
* @return the offset for all the groups of the claimed replication queue,
null means someone else
* has already claimed the queue.
*/
Map<String, ReplicationGroupOffset> claimQueue(String peerId,
ReplicationQueueId queueId,
ServerName targetServerName) throws ReplicationException;
/**
* Remove a replication queue.
* @param queueId the id for the replication queue
*/
void removeQueue(ReplicationQueueId queueId) throws ReplicationException;
{code}
There are some design changes here. One important thing is that, there is no
group in the replication queue id, it is still constructed by a server name and
a peer id. When getting the replication offsets, we will return the replication
set of all the groups. The replication offsets for a queue will be stored in
one row.
The ReplicationSyncUp tool will be broken. It breaks a lot of assumptions in
our design so it is not easy to make it work for now. Will consider it later.
And the implementation of ClaimReplicationQueuesProcedure is not as expected.
In general, maybe we do not need to list all the old wal files, but we need to
list all the replication peers, and create a replication queue for each of them
if not presented. I was thinking to add a replication queue when adding the
peer,but considering a region server start up, you will find out that, you need
to add a queue, i.e, write something to the replication queue storage, for all
the existing peers, when starting a region server. This will introduce cyclic
dependencies between starting a region server and make the replication queue
table online...
Not sure if this will cause some problems, for example, a replication queue is
added after the region server is crashed, but we still create a replication
queue for it and cause the wals to be replicated to peer cluster...
Will consider more when implementing the POC.
Thanks.
> Move HBase replication tracking from ZooKeeper to HBase
> -------------------------------------------------------
>
> Key: HBASE-15867
> URL: https://issues.apache.org/jira/browse/HBASE-15867
> Project: HBase
> Issue Type: New Feature
> Components: Replication
> Affects Versions: 2.1.0
> Reporter: Joseph
> Assignee: Zheng Hu
> Priority: Major
>
> Move the WAL file and offset tracking out of ZooKeeper and into an HBase
> table called hbase:replication.
> The largest three new changes will be two classes ReplicationTableBase,
> TableBasedReplicationQueues, and TableBasedReplicationQueuesClient. As of now
> ReplicationPeers and HFileRef's tracking will not be implemented. Subtasks
> have been filed for these two jobs.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
