Thanks Enrico. This was not needed with Bookkeeper 4.7.3 and without any init/metaformat we were able to bring up the bookie. What has changed in 4.9.1 because of which this is needed now?
-Thanks, Prajakta From: Enrico Olivelli <eolive...@gmail.com> Sent: Thursday, October 10, 2019 2:06 PM To: user Cc: Flavio Junqueira Subject: Re: Error starting bookie using Apache Bookkeeper 4.9.1 [EXTERNAL EMAIL] Looking at the code I think that the short answer is "yes" https://github.com/apache/bookkeeper/blob/2f996dcf0159f945f7ec97ce7402e5d293009444/bookkeeper-server/src/main/java/org/apache/bookkeeper/discover/ZKRegistrationManager.java#L403 we are using ZooKeeper#create, that does not recursively create all the hierarchy. I feel this is due to the fact that usually a BooKeeper cluster stores metadata in /ledgers, as a direct child of '/' IIRC in some product of my company we had the same problem and we simply pre-created the znode structure before '/ledgers' Maybe you can give it a try. I also think it is worth to submit a simply patch in order to change the behaviour and create the full hierarchy (the change is really simple) Hope that helprs Enrico Il giorno gio 10 ott 2019 alle ore 10:28 <prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>> ha scritto: Does that need to be created manually? From: Enrico Olivelli <eolive...@gmail.com<mailto:eolive...@gmail.com>> Sent: Thursday, October 10, 2019 12:25 PM To: user Cc: Flavio Junqueira Subject: Re: Error starting bookie using Apache Bookkeeper 4.9.1 [EXTERNAL EMAIL] I see your base path is /pravega/pravega/bookkeeper/ledgers Maybe the root path does not yet exist, I mean /pravega/pravega/bookkeeper Enrico Il giorno gio 10 ott 2019 alle ore 08:20 <prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>> ha scritto: Thanks for your response, Enrico. I tried using both `metaformat` and `initnewcluster` shell commands to initialize zookeeper before starting the bookie. But both failed with similar exceptions…. Using `bin/bookkeeper shell metaformat -nonInteractive -force` I see … “ Exception in thread "main" java.util.concurrent.ExecutionException: KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378) at org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1150) at org.apache.bookkeeper.bookie.BookieShell$MetaFormatCmd.runCmd(BookieShell.java:328) at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277) at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081) at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:711) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:705) at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.create(ZooKeeperClient.java:705) at org.apache.bookkeeper.discover.ZKRegistrationManager.prepareFormat(ZKRegistrationManager.java:403) at org.apache.bookkeeper.client.BookKeeperAdmin.lambda$format$2(BookKeeperAdmin.java:1152) at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373) “ Using ` bin/bookkeeper shell initnewcluster` I see: “ Exception in thread "main" java.util.concurrent.ExecutionException: KeeperErrorCode = NoNode at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378) at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithRegistrationManager(MetadataDrivers.java:398) at org.apache.bookkeeper.client.BookKeeperAdmin.initNewCluster(BookKeeperAdmin.java:1197) at org.apache.bookkeeper.bookie.BookieShell$InitNewCluster.runCmd(BookieShell.java:368) at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277) at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081) at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:1015) at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:919) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$801(ZooKeeperClient.java:70) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$2.call(ZooKeeperClient.java:466) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$2.call(ZooKeeperClient.java:455) at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.multi(ZooKeeperClient.java:455) at org.apache.bookkeeper.discover.ZKRegistrationManager.initNewCluster(ZKRegistrationManager.java:453) at org.apache.bookkeeper.client.BookKeeperAdmin.lambda$initNewCluster$3(BookKeeperAdmin.java:1199) at org.apache.bookkeeper.meta.MetadataDrivers.lambda$runFunctionWithRegistrationManager$1(MetadataDrivers.java:398) at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373) “ What initialization step am I missing that needs to be done before invoking: ` /opt/bookkeeper/scripts/entrypoint.sh bookie` ?? -Thanks, Prajakta From: Enrico Olivelli <eolive...@gmail.com<mailto:eolive...@gmail.com>> Sent: Friday, October 4, 2019 9:09 PM To: user@bookkeeper.apache.org<mailto:user@bookkeeper.apache.org> Cc: f...@apache.org<mailto:f...@apache.org> Subject: Re: Error starting bookie using Apache Bookkeeper 4.9.1 [EXTERNAL EMAIL] Did you format/init the cluster? Enrico Il ven 4 ott 2019, 16:33 <prajakta.belgu...@dell.com<mailto:prajakta.belgu...@dell.com>> ha scritto: Hi Bookkeeper Team, I’m a Pravega developer and I’m currently trying to use the recent Bookkeeper version `4.9.1` with Pravega (docker image apache/bookkeeper:4.9.1<https://hub.docker.com/r/apache/bookkeeper/tags>) However, when starting a new bookie, I see this exception: “ 2019-10-03 14:53:43,836 - INFO - [main-EventThread:ZooKeeperWatcherBase@130] - ZooKeeper client is connected now. 2019-10-03 14:53:43,860 - ERROR - [main:ZKRegistrationManager@379] - BookKeeper metadata doesn't exist in zookeeper. Has the cluster been initialized? Try running bin/bookkeeper shell metaformat 2019-10-03 14:53:43,861 - INFO - [main:BookieNettyServer@424] - Shutting down BookieNettyServer 2019-10-03 14:53:43,875 - ERROR - [main:Main@223] - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$MetadataStoreException: Failed to get cluster instance id at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:392) at org.apache.bookkeeper.bookie.Bookie.checkEnvironmentWithStorageExpansion(Bookie.java:408) at org.apache.bookkeeper.bookie.Bookie.checkEnvironment(Bookie.java:252) at org.apache.bookkeeper.bookie.Bookie.<init>(Bookie.java:691) at org.apache.bookkeeper.proto.BookieServer.newBookie(BookieServer.java:137) at org.apache.bookkeeper.proto.BookieServer.<init>(BookieServer.java:106) at org.apache.bookkeeper.server.service.BookieService.<init>(BookieService.java:43) at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:301) at org.apache.bookkeeper.server.Main.doMain(Main.java:221) at org.apache.bookkeeper.server.Main.main(Main.java:203) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for BookKeeper metadata at org.apache.bookkeeper.discover.ZKRegistrationManager.getClusterInstanceId(ZKRegistrationManager.java:382) “ If I try to run `metaformat` or `initnewCluster` using the bookkeeper shell, before starting the new bookie, that too fails because the expected znode is not present… Error with `metaformat`: “ Exception in thread "main" java.util.concurrent.ExecutionException: KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:378) at org.apache.bookkeeper.client.BookKeeperAdmin.format(BookKeeperAdmin.java:1150) at org.apache.bookkeeper.bookie.BookieShell$MetaFormatCmd.runCmd(BookieShell.java:328) at org.apache.bookkeeper.bookie.BookieShell$MyCommand.runCmd(BookieShell.java:277) at org.apache.bookkeeper.bookie.BookieShell.run(BookieShell.java:3081) at org.apache.bookkeeper.bookie.BookieShell.main(BookieShell.java:3172) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /pravega/pravega/bookkeeper/ledgers at org.apache.zookeeper.KeeperException.create(KeeperException.java:114) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$1901(ZooKeeperClient.java:70) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:711) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$9.call(ZooKeeperClient.java:705) at org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140) at org.apache.bookkeeper.zookeeper.ZooKeeperClient.create(ZooKeeperClient.java:705) at org.apache.bookkeeper.discover.ZKRegistrationManager.prepareFormat(ZKRegistrationManager.java:403) at org.apache.bookkeeper.client.BookKeeperAdmin.lambda$format$2(BookKeeperAdmin.java:1152) at org.apache.bookkeeper.meta.MetadataDrivers.runFunctionWithMetadataBookieDriver(MetadataDrivers.java:373 ” Could you please help me understand what could be causing this? -Thanks, Prajakta
