[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397072683 Thanks for your help..I shall keep you posted for any further questions. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397069871 Brokers seems to be working now and able to discover the bookies configured 13:01:31.329 [bookkeeper-io-12-11] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host5:3181 13:01:31.332 [bookkeeper-io-12-17] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: host4:3181 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397063453 Bookies came up fine after above changes and I see hostname configured for bookie host. Do you recommend any changes on Brokers conf other than advertisedAddress before I restart the same ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397000397 Ledger version: [root@host5 current]# cat VERSION 4 bookieHost: "2001::a06::f1:fef:0:::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396973645 When bookie format is attempted, I can't start back the bookie process again since it fails with the same error report above and data directories are empty. **Working Bookie Node version with useHostNameAsBookieID set to False.** bookieHost: "2001::a06::f1:fef:0::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" And by default only below directories are created and no storage nodes [root@host5 bookkeeper]# ls -lrt total 8 drwx-- 3 root root 4096 Jun 12 11:50 journal drwx-- 3 root root 4096 Jun 12 11:50 ledgers This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396795541 Attached logs for your reference [bookkeeper.conf.txt](https://github.com/apache/incubator-pulsar/files/2099500/bookkeeper.conf.txt) [pulsar-bookie-host4.log](https://github.com/apache/incubator-pulsar/files/2099502/pulsar-bookie-host4.log) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396784882 Also when useHostNameAsBookieID is set to true and bookie restarted , process fails with the below error. https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97 The only way I could get the sanity passed is when the above setting is false , bookie started and sanity run with bookie conf file updated 18:22:49.968 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 18:22:50.095 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 18:22:50.106 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "host5:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" ] is not matching with [4 bookieHost: "2001::a06::f1:fef:0::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396757371 All the bookies are up . Below is the broker conf update # Zookeeper quorum connection string zookeeperServers=host8:2181,host9:2181,host10:2181 # Configuration Store connection string configurationStoreServers=host8v:2181,host9:2181,host10:2181 advertisedAddress=host1 (Broker hostname) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396742632 @sijie useHostNameAsBookieID to true (worked fine) but having issues with brokers 14:34:08.223 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x9faf4b3a, L:/2001::a06::f1:fef:0:dgdg%0:5317 - R:host1/2001::a06::f1:fef:0::6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.224 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.225 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages 14:34:08.226 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397045930 Version file still uses IPV6 since bookie never started successfully after modifying conf file This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-397000397 Ledger version: [root@carocnxodapa5v current]# cat VERSION 4 bookieHost: "2001::a06::f1:fef:0:::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396973645 When bookie format is attempted, I can't start back the bookie process again since it fails with the same error report above and data directories are empty. **Working Bookie Node version with useHostNameAsBookieID set to False.** bookieHost: "2001::a06::f1:fef:0::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" And by default only below directories are created and no storage nodes [root@carocnxodapa5v bookkeeper]# ls -lrt total 8 drwx-- 3 root root 4096 Jun 12 11:50 journal drwx-- 3 root root 4096 Jun 12 11:50 ledgers This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396795541 Attached logs for your reference [bookkeeper.conf.txt](https://github.com/apache/incubator-pulsar/files/2096426/bookkeeper.conf.txt) [pulsar-bookie-carocnxodapa4v.log](https://github.com/apache/incubator-pulsar/files/2096436/pulsar-bookie-carocnxodapa4v.log) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396795541 Attached logs for your reference [bookkeeper.conf.txt](https://github.com/apache/incubator-pulsar/files/2096426/bookkeeper.conf.txt) [pulsar-bookie-carocnxodapa4v.log](https://github.com/apache/incubator-pulsar/files/2096427/pulsar-bookie-carocnxodapa4v.log) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396793056 Bookie doesn't start .. 19:22:42.675 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 19:22:42.702 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "carocnxodapa4v:3181" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396789828 Fails with the same error after "bookieformat" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396789524 19:00:27.594 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 19:00:27.702 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer Fails with the same error 19:00:27.722 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "carocnxodapa4v:3181" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396788417 Should I restart bookie after bookieformat? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396784882 Also when useHostNameAsBookieID is set to true and bookie restarted , process fails with the below error. https://github.com/apache/incubator-pulsar/blob/master/conf/bookkeeper.conf#L97 The only way I could get the sanity passed is when the above setting is false , bookie started and sanity run with bookie conf file updated 18:22:49.968 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 18:22:50.095 [main] INFO org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer 18:22:50.106 [main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "carocnxoappa5v:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" ] is not matching with [4 bookieHost: "2001::a06::f1:fef:0::3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "b5961471-638e-46b0-b537-51bdf5a1f116" This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396781464 Unfortunately nothing on port 3181..Is my configuration above for starting broker correct? I have 3 zookeepers and 3 bookies running ..For some strange reason brokers aren't able to see them This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396757371 All the bookies are up . Below is the broker conf update # Zookeeper quorum connection string zookeeperServers=carocnxoappa8v:2181,carocnxoappa9v:2181,carocnxoappa10v:2181 # Configuration Store connection string configurationStoreServers=carocnxoappa8v:2181,carocnxoappa9v:2181,carocnxoappa10:2181 advertisedAddress=carocnxodapa1 (Broker hostname) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396755923 Tried ./bin/bookkeeper shell listbookies -rw and it shows up lot of logs ..do you need them ? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396752764 I am running 3 bookies ...how should I run the below command [root@carocnxodapa1v apache-pulsar-2.0.0-rc1-incubating]# ./bin/bookkeeper listbookies -rw JMX enabled by default JAVA_HOME not set, using java from PATH. (/usr/bin/java) Error: Could not find or load main class listbookies This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger
dsambandam commented on issue #1953: Bookie sanity fails - Unable to connect to Bookie after writing to Ledger URL: https://github.com/apache/incubator-pulsar/issues/1953#issuecomment-396742632 @sijie useHostNameAsBookieID to true (worked fine) but having issues with brokers 14:34:08.223 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0x9faf4b3a, L:/2001:4888:a06:2281:f1:fef:0:c49a%0:5317 - R:carocnxodapa1v/2001::a06::f1:fef:0::6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.224 [pulsar-client-io-1-1] ERROR org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Failed to create producer: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available 14:34:08.225 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages 14:34:08.226 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available org.apache.pulsar.client.api.PulsarClientException$BrokerPersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
