[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed the addendum that will help if a hang. > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 0001-HBASE-19694.addendum.branch-2.patch, > HBASE-19694.branch-2.001.patch, HBASE-19694.branch-2.002.patch, > HBASE-19694.branch-2.003.patch, HBASE-19694.branch-2.004.patch, > HBASE-19694.branch-2.005.patch, HBASE-19694.branch-2.006.patch, > HBASE-19694.branch-2.007.patch, HBASE-19694.branch-2.008.patch, > HBASE-19694.branch-2.009.patch, HBASE-19694.branch-2.010.patch, > HBASE-19694.branch-2.011.patch, HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Status: Patch Available (was: Reopened) Reopen to try my addendum. > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 0001-HBASE-19694.addendum.branch-2.patch, > HBASE-19694.branch-2.001.patch, HBASE-19694.branch-2.002.patch, > HBASE-19694.branch-2.003.patch, HBASE-19694.branch-2.004.patch, > HBASE-19694.branch-2.005.patch, HBASE-19694.branch-2.006.patch, > HBASE-19694.branch-2.007.patch, HBASE-19694.branch-2.008.patch, > HBASE-19694.branch-2.009.patch, HBASE-19694.branch-2.010.patch, > HBASE-19694.branch-2.011.patch, HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: 0001-HBASE-19694.addendum.branch-2.patch Small addendum. Running it to see state of build on tip of branch-2. > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: 0001-HBASE-19694.addendum.branch-2.patch, > HBASE-19694.branch-2.001.patch, HBASE-19694.branch-2.002.patch, > HBASE-19694.branch-2.003.patch, HBASE-19694.branch-2.004.patch, > HBASE-19694.branch-2.005.patch, HBASE-19694.branch-2.006.patch, > HBASE-19694.branch-2.007.patch, HBASE-19694.branch-2.008.patch, > HBASE-19694.branch-2.009.patch, HBASE-19694.branch-2.010.patch, > HBASE-19694.branch-2.011.patch, HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master and branch-2 > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch, HBASE-19694.branch-2.009.patch, > HBASE-19694.branch-2.010.patch, HBASE-19694.branch-2.011.patch, > HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.012.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch, HBASE-19694.branch-2.009.patch, > HBASE-19694.branch-2.010.patch, HBASE-19694.branch-2.011.patch, > HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.011.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch, HBASE-19694.branch-2.009.patch, > HBASE-19694.branch-2.010.patch, HBASE-19694.branch-2.011.patch, > HBASE-19694.branch-2.012.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.010.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch, HBASE-19694.branch-2.009.patch, > HBASE-19694.branch-2.010.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.009.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch, HBASE-19694.branch-2.009.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.008.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch, > HBASE-19694.branch-2.008.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.007.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch, HBASE-19694.branch-2.007.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.006.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch, > HBASE-19694.branch-2.006.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.005.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch, HBASE-19694.branch-2.005.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.004.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch, > HBASE-19694.branch-2.004.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.003.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch, HBASE-19694.branch-2.003.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.002.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch, > HBASE-19694.branch-2.002.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Status: Patch Available (was: Open) > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Fix Version/s: (was: 2.0.0-beta-2) 2.0.0-beta-1 > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-1 > > Attachments: HBASE-19694.branch-2.001.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Attachment: HBASE-19694.branch-2.001.patch > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Assignee: stack >Priority: Critical > Fix For: 2.0.0-beta-2 > > Attachments: HBASE-19694.branch-2.001.patch > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Fix Version/s: 2.0.0-beta-2 > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HBASE-19694) The initialization order for a fresh cluster is incorrect
[ https://issues.apache.org/jira/browse/HBASE-19694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19694: -- Priority: Critical (was: Major) > The initialization order for a fresh cluster is incorrect > - > > Key: HBASE-19694 > URL: https://issues.apache.org/jira/browse/HBASE-19694 > Project: HBase > Issue Type: Bug >Reporter: Duo Zhang >Priority: Critical > Fix For: 2.0.0-beta-2 > > > The cluster id will set once we become the active master in > finishActiveMasterInitialization, but the blockUntilBecomingActiveMaster and > finishActiveMasterInitialization are both called in a thread to make the > constructor of HMaster return without blocking. And since HMaster itself is > also a HRegionServer, it will create a Connection and then start calling > reportForDuty. And when creating the ConnectionImplementation, we will read > the cluster id from zk, but the cluster id may have not been set yet since it > is set in another thread, we will get an exception and use the default > cluster id instead. > I always get this when running UTs which will start a mini cluster > {noformat} > 2018-01-03 15:16:37,916 WARN [M:0;zhangduo-ubuntu:32848] > client.ConnectionImplementation(528): Retrieve cluster id failed > java.util.concurrent.ExecutionException: > org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = > NoNode for /hbase/hbaseid > at > java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) > at > java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.retrieveClusterId(ConnectionImplementation.java:526) > at > org.apache.hadoop.hbase.client.ConnectionImplementation.(ConnectionImplementation.java:286) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:141) > at > org.apache.hadoop.hbase.client.ConnectionUtils$ShortCircuitingClusterConnection.(ConnectionUtils.java:137) > at > org.apache.hadoop.hbase.client.ConnectionUtils.createShortCircuitConnection(ConnectionUtils.java:185) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.createClusterConnection(HRegionServer.java:781) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.setupClusterConnection(HRegionServer.java:812) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:827) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:938) > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:550) > at java.lang.Thread.run(Thread.java:748) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /hbase/hbaseid > at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:163) > at > org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:311) > ... 1 more > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)