[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[
https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17372142#comment-17372142
]
Hudson commented on HBASE-26029:
Results for branch branch-2
[build #289 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/289//console].
> It is not reliable to use nodeDeleted event to track region server's death
> --
>
> Key: HBASE-26029
> URL: https://issues.apache.org/jira/browse/HBASE-26029
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario,
> where a region server up and down between two sync requests, then we can not
> know the death of the region server.
> https://github.com/apache/hbase/pull/3405#discussion_r656720923
> This is a valid point, and when thinking of a solution, I noticed that, the
> current zk iplementation has the same problem. Notice that, a watcher on zk
> can only be triggered once, so after zk triggers the watcher, and before you
> set a new watcher, it is possible that a region server is up and down, and
> you will miss the nodeDeleted event for this region server.
> I think, the general approach here, which could works for both master based
> and zk based replication tracker is that, we should not rely on the tracker
> to tell you which region server is dead. Instead, we just provide the list of
> live regionservers, and the upper layer should compare this list with the
> expected list(for replication, the list should be gotten by listing
> replicators), to detect the dead region servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[
https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371994#comment-17371994
]
Hudson commented on HBASE-26029:
Results for branch master
[build #335 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/master/335/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> It is not reliable to use nodeDeleted event to track region server's death
> --
>
> Key: HBASE-26029
> URL: https://issues.apache.org/jira/browse/HBASE-26029
> Project: HBase
> Issue Type: Bug
> Components: Replication, Zookeeper
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario,
> where a region server up and down between two sync requests, then we can not
> know the death of the region server.
> https://github.com/apache/hbase/pull/3405#discussion_r656720923
> This is a valid point, and when thinking of a solution, I noticed that, the
> current zk iplementation has the same problem. Notice that, a watcher on zk
> can only be triggered once, so after zk triggers the watcher, and before you
> set a new watcher, it is possible that a region server is up and down, and
> you will miss the nodeDeleted event for this region server.
> I think, the general approach here, which could works for both master based
> and zk based replication tracker is that, we should not rely on the tracker
> to tell you which region server is dead. Instead, we just provide the list of
> live regionservers, and the upper layer should compare this list with the
> expected list(for replication, the list should be gotten by listing
> replicators), to detect the dead region servers.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17371044#comment-17371044 ] Bharath Vissapragada commented on HBASE-26029: -- Hooking this up with SCP is neat, should clean up all the dead server management code from ReplicationSourceManager, +1. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-26029) It is not reliable to use nodeDeleted event to track region server's death
[ https://issues.apache.org/jira/browse/HBASE-26029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17368830#comment-17368830 ] Duo Zhang commented on HBASE-26029: --- After talkling with [~sunxin] offline, we think that the most clear way to fix this, is to add a step for SCP to assign the replication queue of the dead server to different region servers. > It is not reliable to use nodeDeleted event to track region server's death > -- > > Key: HBASE-26029 > URL: https://issues.apache.org/jira/browse/HBASE-26029 > Project: HBase > Issue Type: Bug > Components: Replication, Zookeeper >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > > When implementing HBASE-26011, [~sunxin] pointed out an interesting scenario, > where a region server up and down between two sync requests, then we can not > know the death of the region server. > https://github.com/apache/hbase/pull/3405#discussion_r656720923 > This is a valid point, and when thinking of a solution, I noticed that, the > current zk iplementation has the same problem. Notice that, a watcher on zk > can only be triggered once, so after zk triggers the watcher, and before you > set a new watcher, it is possible that a region server is up and down, and > you will miss the nodeDeleted event for this region server. > I think, the general approach here, which could works for both master based > and zk based replication tracker is that, we should not rely on the tracker > to tell you which region server is dead. Instead, we just provide the list of > live regionservers, and the upper layer should compare this list with the > expected list(for replication, the list should be gotten by listing > replicators), to detect the dead region servers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
