[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Fix Version/s: 2.1.0
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master-v2.patch,
> HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Merged to master.
Thanks all for contributing and reviewing.
Will open follow on issue to integrate into branch-2.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Duo Zhang
> Fix For: 3.0.0
>
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master-v2.patch,
> HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Issue Type: New Feature (was: Improvement)
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Duo Zhang
> Fix For: 3.0.0
>
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master-v2.patch,
> HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master-v2.patch
Rebased.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master-v2.patch,
> HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-branch-2.patch
Patch for branch-2.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-branch-2.patch, HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignored since we have
already done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
was:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master-v1.patch
Retry.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch,
> HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
On rolling upgrade: just do not do any replication peer modifications during
the rolling upgrading. There is no pb/layout changes on the peer/queue storage
on zk.
was:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master-v1.patch
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master-v1.patch, HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397-master.patch
OK, we need to include a 'master' in the name of the patch...
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397-master.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: (was: HBASE-19397.patch)
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Status: Patch Available (was: Open)
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Attachment: HBASE-19397.patch
Merge all the commits into a single patch.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
> Attachments: HBASE-19397.patch
>
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now it is guaranteed that once the procedure is done, the peer modification
has already taken effect on all RSes.
Abstracte a storage layer for replication peer/queue manangement, and
refactored the upper layer to remove zk related naming/code/comment.
Add pre/postExecuteProcedures CP hooks to RegionServerObserver, and add
permission check for executeProcedures method which requires the caller to be
system user or super user.
was:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now we can guarantee that once the procedure is done, the peer modification
has already taken effect on all RSes.
We have also abstracted a storage layer for replication peer/queue manangement,
and refactored the upper layer to remove zk related naming/code/comment.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Release Note:
Introduce 5 procedures to do peer modifications:
AddPeerProcedure
RemovePeerProcedure
UpdatePeerConfigProcedure
EnablePeerProcedure
DisablePeerProcedure
The procedures are all executed with the following stage:
1. Call pre CP hook, if an exception is thrown then give up
2. Check whether the operation is valid, if not then give up
3. Update peer storage. Notice that if we have entered this stage, then we can
not rollback any more.
4. Schedule sub procedures to refresh the peer config on every RS.
5. Do post cleanup if any.
6. Call post CP hook. The exception thrown will be ignore since we have already
done the work.
The procedure will hold an exclusive lock on the peer id, so now there is no
concurrent modifications on a single peer.
And now we can guarantee that once the procedure is done, the peer modification
has already taken effect on all RSes.
We have also abstracted a storage layer for replication peer/queue manangement,
and refactored the upper layer to remove zk related naming/code/comment.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Component/s: proc-v2
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: proc-v2, Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-19397:
--
Issue Type: New Feature (was: Sub-task)
Parent: (was: HBASE-15867)
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HBASE-19397) Design procedures for ReplicationManager to notify peer change event from master
[
https://issues.apache.org/jira/browse/HBASE-19397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zheng Hu updated HBASE-19397:
-
Description:
After we store peer states / peer queues information into hbase table, RS can
not track peer config change by adding watcher znode.
So we need design procedures for ReplicationManager to notify peer change
event. the replication rpc interfaces which may be implemented by procedures
are following:
{code}
1. addReplicationPeer
2. removeReplicationPeer
3. enableReplicationPeer
4. disableReplicationPeer
5. updateReplicationPeerConfig
{code}
BTW, our RS states will still be store in zookeeper, so when RS crash, the
tracker which will trigger to transfer queues of crashed RS will still be a
Zookeeper Tracker. we need NOT implement that by procedures.
As we will release 2.0 in next weeks, and the HBASE-15867 can not be resolved
before the release, so I'd prefer to create a new feature branch for
HBASE-15867.
was:
After we store peer states / peer queues information into hbase table, RS can
not tracker peer config change by adding watcher znode.
So we need design procedures for ReplicationManager to notify peer change
event. the replication rpc interfaces which may be implemented by procedures
are following:
{code}
1. addReplicationPeer
2. removeReplicationPeer
3. enableReplicationPeer
4. disableReplicationPeer
5. updateReplicationPeerConfig
{code}
BTW, our RS states will still be store in zookeeper, so when RS crash, the
tracker which will trigger to transfer queues of crashed RS will still be a
Zookeeper Tracker. we need NOT implement that by procedures.
As we will release 2.0 in next weeks, and the HBASE-15867 can not be resolved
before the release, so I'd prefer to create a new feature branch for
HBASE-15867.
> Design procedures for ReplicationManager to notify peer change event from
> master
> -
>
> Key: HBASE-19397
> URL: https://issues.apache.org/jira/browse/HBASE-19397
> Project: HBase
> Issue Type: Sub-task
> Components: Replication
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>
> After we store peer states / peer queues information into hbase table, RS
> can not track peer config change by adding watcher znode.
> So we need design procedures for ReplicationManager to notify peer change
> event. the replication rpc interfaces which may be implemented by
> procedures are following:
> {code}
> 1. addReplicationPeer
> 2. removeReplicationPeer
> 3. enableReplicationPeer
> 4. disableReplicationPeer
> 5. updateReplicationPeerConfig
> {code}
> BTW, our RS states will still be store in zookeeper, so when RS crash, the
> tracker which will trigger to transfer queues of crashed RS will still be a
> Zookeeper Tracker. we need NOT implement that by procedures.
> As we will release 2.0 in next weeks, and the HBASE-15867 can not be
> resolved before the release, so I'd prefer to create a new feature branch
> for HBASE-15867.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
