[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15615196#comment-15615196 ] Hudson commented on HBASE-16835: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1870 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1870/]) HBASE-16835 Revisit the zookeeper usage at client side (zhangduo: rev 3283bc7c91b61b9687ba6f98e1a28ef6ab81e432) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableNoncedRetry.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestZKAsyncRegistry.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterRegistryFactory.java * (delete) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterRegistry.java * (delete) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKClusterRegistry.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/RecoverableZooKeeper.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncRegistry.java * (edit) hbase-client/pom.xml * (edit) pom.xml * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncConnectionImpl.java > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835-v1.patch, HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614324#comment-15614324 ] stack commented on HBASE-16835: --- No. Looks great. +1. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835-v1.patch, HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614314#comment-15614314 ] Duo Zhang commented on HBASE-16835: --- Any other concerns of the new patch? [~stack] Thanks. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835-v1.patch, HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614275#comment-15614275
]
Hadoop QA commented on HBASE-16835:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 38s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
40s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 0s
{color} | {color:red} hbase-client in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 26s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 16s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 29s
{color} | {color:red} root in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 0s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 35s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 154m 42s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestAdmin2 |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestSnapshotFromClientWithRegionReplicas |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15613886#comment-15613886 ] Duo Zhang commented on HBASE-16835: --- Fine. Let et change the class name. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612433#comment-15612433 ] stack commented on HBASE-16835: --- Thanks [~busbey] Filed HBASE-16955. Let me have a go at it. Shouldn't be too bad after move to protobuf-maven-plugin which manages its own protoc when pb3.1.0. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612410#comment-15612410 ] stack commented on HBASE-16835: --- Thanks for clarification. Seems odd to call it ClusterRegistry. Registry describes itself as a Cluster Registry. So does Cluster Registry. Would AsyncRegistry be better name? > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612387#comment-15612387
]
Sean Busbey commented on HBASE-16835:
-
{quote}
bq. We don't do protoc as part of our build. It is a step done outside of build
to generate src that is checked in for use by build.
So we need to change the precommit job config? Sean Busbey What's purpose of
running protoc check in precommit?
{quote}
checking that protoc builds still work after a change was added in 2014 by
[~tedyu] and [~apurtell] in HBASE-11375 after we had some changes come in that
broke doing a protoc build. The fact that we don't usually do a protoc run when
building makes it more important to check in precommit (IMHO) because that
means we won't discover the problem until far after the change goes in.
Now that we're doing gymnastics around protobuf, we should probably make sure
we can do end-to-end regeneration of all the protobuf stuff in our precommit
job, since we have the advantage of requiring a consistent build environment
(via docker).
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
> Components: Client, Zookeeper
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16835.patch
>
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611081#comment-15611081 ] Duo Zhang commented on HBASE-16835: --- Registry is design for blocking access only and the new ClusterRegistry is design for async access. It returns CompletableFuture for most methods. And the zooKeeperRegistry depends on ConnectionImplementation so we can not use it directly and modify it may effect the blocking client implementation. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15611058#comment-15611058 ] stack commented on HBASE-16835: --- RIght. We have Registry and ClusterRegistry. Why out put the Registry methods into ClusterRegistry? > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610992#comment-15610992 ] Duo Zhang commented on HBASE-16835: --- We do have a Registry in master... https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/Registry.java > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610963#comment-15610963 ] stack commented on HBASE-16835: --- When you say old registry, you mean branch-1? Yeah, precommit job might need some fixup now we have protos distributed and we have different protocs needed. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610908#comment-15610908
]
Duo Zhang commented on HBASE-16835:
---
{quote}
We don't do protoc as part of our build. It is a step done outside of build to
generate src that is checked in for use by build.
{quote}
So we need to change the precommit job config? [~busbey] What's purpose of
running protoc check in precommit?
{quote}
Do another Interface for your new stuff?
{quote}
The old Registry interface has three methods, getMetaRegionLocation,
getClusterId and getCurrentNrHRS. The data are fetched from zk. I add another
two methods to the new interface as the master address is also stored on zk.
All zk related things at client side are placed in this interface.
Thanks.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
> Components: Client, Zookeeper
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16835.patch
>
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610882#comment-15610882
]
stack commented on HBASE-16835:
---
bq. But I do not think the precommit job could read our document and change the
default protoc when building HBase...
We don't do protoc as part of our build. It is a step done outside of build to
generate src that is checked in for use by build.
bq. Is it possible to make use of protobuf-maven-plugin when building
hbase-protocol-shaded?
Thanks for the pointer. It is better than the plugin we were using. Let me use
it everywhere w/ clear instruction that it will take care of using right protoc
in hbase-protocol-shaded but everywhere else you need to supply the pb2.5.0
since it cant. It is better in that I do not have to list the .protos in two
places, in the dir and in the pom.
On the patch, aren't you polluting ClusterRegistry interface when you add those
new methods that don't seem to have anything to do w/ Cluster ("Should only be
called once."... then closed)? Do another Interface for your new stuff?
Otherwise patch looks good to me. Trying to see if it could be incompat with an
older client writing zk but not seeing it.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
> Components: Client, Zookeeper
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16835.patch
>
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610317#comment-15610317 ] Duo Zhang commented on HBASE-16835: --- But I do not think the precommit job could read our document and change the default protoc when building HBase... Is it possible to make use of [protobuf-maven-plugin|https://www.xolstice.org/protobuf-maven-plugin/] when building hbase-protocol-shaded? It can download the protoc artifact(starting from version 2.6.1) from the maven repo. And, any concerns of the patch here? [~stack] Thanks. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15609809#comment-15609809 ] stack commented on HBASE-16835: --- Pushed HBASE-16949 to fix the above. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608868#comment-15608868 ] Anoop Sam John commented on HBASE-16835: bq.hbase-protocol-shaded/src/main/patches/HBASE-15789_V2.patch Ya we should exclude that from rat check. And make sure these wont go out as part of the src tarball. Thanks for bringing this up. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608307#comment-15608307 ] Duo Zhang commented on HBASE-16835: --- [~stack] The patch you committed in hbase-15789 missed the asf header. Or we should exclude it from the rat check? This file hbase-protocol-shaded/src/main/patches/HBASE-15789_V2.patch And for protoc, we need protoc 2.5.0 for hbase-protocol and protoc 3.1.0 for hbase-protocol-shaded? We need some tricks in the pom.xml I think... > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task > Components: Client, Zookeeper >Affects Versions: 2.0.0 >Reporter: Duo Zhang >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-16835.patch > > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608064#comment-15608064
]
Hadoop QA commented on HBASE-16835:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
33s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
39s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 34s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha1. {color} |
| {color:red}-1{color} | {color:red} hbaseprotoc {color} | {color:red} 0m 21s
{color} | {color:red} root in the patch failed. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 9s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 37s {color}
| {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 40s
{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestCloneSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835285/HBASE-16835.patch |
| JIRA Issue | HBASE-16835 |
| Optional Tests | asflicense javac javadoc unit xml compile findbugs
hadoopcheck hbaseanti checkstyle |
|
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607773#comment-15607773
]
Hadoop QA commented on HBASE-16835:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16835 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12835283/HBASE-16835.patch |
| JIRA Issue | HBASE-16835 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4191/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
> Components: Client, Zookeeper
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16835.patch
>
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15580409#comment-15580409 ] stack commented on HBASE-16835: --- bq. Yeah but I think we still need to fetch the root region location from zk Didn't want to go there (smile). It would be easy to do a tidier zk layout. There are a few tools that read zk directly to figure stuff but we can deprecate the old znodes and force a move to the new APIs asking master for info they need instead. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579893#comment-15579893 ] Duo Zhang commented on HBASE-16835: --- Will come back when I finish the async scan since we can not implement async region locator without scan. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15579875#comment-15579875
]
Duo Zhang commented on HBASE-16835:
---
{quote}
Could we add the web ui port and other vitals to the master znode data. It is
pb so extensible.
{quote}
They have already been in the same pb message. getMasterAddress and
getMasterInfoPort actually get the same node from zk but retrieve the different
part of the data. Maybe we need a new data structure?
{quote}
The latter will change later when we can split meta table but it is fine for
now.
{quote}
Yeah but I think we still need to fetch the root region location from zk :)
{quote}
You think then that we'd have a new Registry API for async use? Could the new
API underpin the old API?
{quote}
As we will build the old sync API on top of the new async API, I think we'd
better not change the old code too much as it will finally be removed? And for
this case, I want to move all zk related into ClusterRegistry. The old registry
does not have the master address stuff. It is the MasterAddressTracker.
Thanks.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578087#comment-15578087
]
Duo Zhang commented on HBASE-16835:
---
{code:title=ClusterRegistry.java}
interface ClusterRegistry extends Closeable {
RegionLocations getMetaRegionLocation();
/**
* Should only be called once.
*
* The upper layer should store this value somewhere as it will not be change
any more.
*/
String getClusterId();
int getCurrentNrHRS();
ServerName getMasterAddress();
int getMasterInfoPort();
@Override
void close();
}
{code}
This is the original ClusterRegistry interface introduced in HBASE-15921. I
added all the zk related operations needed at client side to it.
The getClusterId will only be called once when start up so it can be blocking
and read directly from zk. And for getCurrentNrHRS, I think it is not very
important so let's discuss it later. The getMasterInfoPort gets the same thing
from zookeeper with getMasterAddress. So actually, the most important methods
are getMetaRegionLocation and getMasterAddress.
For meta location, we will also cache it meta cache. And for master address, we
have a logic to decide when to fetch the new address from zk. And usually, a
client will cache all the region locations needed so it even does not need to
know the meta location change as it will not access the meta table for a long
time. And for master, it is also rarely touched from client. So I think it is
reasonable to not rely on watcher to update the value as it may cause
unnecessary network traffic.
But I think the API of meta cache and master address tracker should be changed.
We have two timeouts right now, one for the normal operations, and the other
one for the meta operations. I think we should have a larger timeout for meta
operations because if we done then we could cache it. But in the old blocking
world, it is a not a good idea to have a larger timeout in a sub stage... In
the asynchronous world, we could return a CompletableFuture and use
HashedWheelTimer to trigger timeout. So it is possible to have a larger timeout
in a sub stage as upper layer has the ability to timeout before the actual
process done.
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[ https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576989#comment-15576989 ] stack commented on HBASE-16835: --- I like the description. Says it well. > Revisit the zookeeper usage at client side > -- > > Key: HBASE-16835 > URL: https://issues.apache.org/jira/browse/HBASE-16835 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang > > Watcher or not. > Curator or not. > Keep connection or not. > ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16835) Revisit the zookeeper usage at client side
[
https://issues.apache.org/jira/browse/HBASE-16835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574258#comment-15574258
]
Duo Zhang commented on HBASE-16835:
---
Paste the comments on rb here.
{quote}
Michael Stack
In the past we avoided curator because our zk thingy -- RecoverableZookeeper --
was supposed to be immune to the case where we'd miss an update. All data in zk
was preceded by some version number and if we did not get the version expected,
then we'd know there'd be a change on us.
Not sure if curator does this now.
This is all stuff from long ago. I'm sure the world changed since but something
to be aware of.
Duo Zhang
ZooKeeper could miss some update, this is by design. Not like etcd, zk does not
give you all the update events of a node. Instead, it just tell you that the
data has been changed, you need to fetch it from zk then. That's why the
watcher can only be notified once. It means when you receive a notification
from watcher then you need to do a getAndWatch or existsAndWatch to get the
result and set a new watcher on it. It does not matter if the data is changed
after the watcher tells you it is changed, you will see the modified data when
you actual get it. And it does not matter if the data is changed after you get
it, you set a watcher, the watcher will tell you the that you need to get the
new data.
I think this is enough for client usage. At the server side, yeah sometimes we
need to get a consistent view of a tree. Curator's PathChildrenCache is not
suitable under this scenario. The only way is to check the version number after
fetching...
Duo Zhang
And I still need curator even if I do not use the recipes in it as our
RecoverableZooKeeper does not support callback. I could add the support for it
but I think it is too heavy for a client since we do not need to write anything
to zookeeper from client...
And I think RS can not recover from a session expire error? But client should
be able to reconnect even after a session expire as we do not store any state
on zk.
Michael Stack
ok. Argument for Curator on the client-side only seems fine, reasonable. How to
ensure client-side only usage. Do I/we need to do a review of zk usage (in
another issue) to identify those cases where we need to see all changes (This
need may be going away w/ our using zk less often)
Duo Zhang
I think this is the duty of committers? We as committer should prevent the
wrong usage introduced by contributors in the future :)
And now I think the only place where we use curator is in the ZKClusterRegistry
introduced in this patch.
Michael Stack
I like your point above that client should be able to recover from a session
expiration. We don't do that currently IIRC. Open an issue so it gets
attention. Good point.
{quote}
> Revisit the zookeeper usage at client side
> --
>
> Key: HBASE-16835
> URL: https://issues.apache.org/jira/browse/HBASE-16835
> Project: HBase
> Issue Type: Sub-task
>Reporter: Duo Zhang
>
> Watcher or not.
> Curator or not.
> Keep connection or not.
> ...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
