EronWright opened a new issue #12764:
URL: https://github.com/apache/pulsar/issues/12764
**Describe the bug**
The broker cannot start when configured with a chroot for its ZK metadata.
**To Reproduce**
Effect a configuration where the brokers and the bookies aren't using the
same ZooKeeper configuration:
```
[conf/broker.conf] Applying config configurationStoreServers =
shared-zk-headless.avengers.svc.cluster.local:2181/PulsarCluster/avengers/hawkeye
[conf/broker.conf] Applying config zookeeperServers =
shared-zk-headless.avengers.svc.cluster.local:2181/PulsarCluster/avengers/hawkeye
[conf/broker.conf] Updating config bookkeeperMetadataServiceUri =
zk+null://shared-zk-headless.avengers.svc.cluster.local:2181/ledgers
```
Verify that the ledger metadata is available:
```
ls /ledgers/available
[readonly, shared-bk-1.shared-bk-headless.avengers.svc.cluster.local:3181,
shared-bk-2.shared-bk-headless.avengers.svc.cluster.local:3181]
```
Observe that the broker cannot startup:
```
2021-11-11T18:48:44,766+0000 [main] ERROR
org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service:
org.apache.bookkeeper.client.BKException$ZKException: Error while using
ZooKeeper
java.io.IOException: org.apache.bookkeeper.client.BKException$ZKException:
Error while using ZooKeeper
at
org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:89)
~[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.pulsar.broker.BookKeeperClientFactoryImpl.create(BookKeeperClientFactoryImpl.java:64)
~[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.pulsar.broker.ManagedLedgerClientFactory.initialize(ManagedLedgerClientFactory.java:86)
~[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.pulsar.broker.storage.ManagedLedgerStorage.create(ManagedLedgerStorage.java:97)
~[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:637)
[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:273)
[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:350)
[io.streamnative-pulsar-broker-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
Caused by: org.apache.bookkeeper.client.BKException$ZKException: Error while
using ZooKeeper
at
org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$4(ZKRegistrationClient.java:351)
~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at
org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$27$1.processResult(PulsarZooKeeperClient.java:1240)
~[io.streamnative-pulsar-metadata-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:678)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for /ledgers/available
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:118)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at
org.apache.bookkeeper.discover.ZKRegistrationClient.lambda$getChildren$4(ZKRegistrationClient.java:350)
~[org.apache.bookkeeper-bookkeeper-server-4.14.2.jar:4.14.2]
at
org.apache.pulsar.metadata.impl.PulsarZooKeeperClient$27$1.processResult(PulsarZooKeeperClient.java:1240)
~[io.streamnative-pulsar-metadata-2.9.0.0-rc-3.jar:2.9.0.0-rc-3]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:678)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:563)
~[org.apache.zookeeper-zookeeper-3.6.3.jar:3.6.3]
```
**Expected behavior**
Normal startup, such that the ledger metadata is drawn from
`bookkeeperMetadataServiceUri` not from `zookeeperServers`.
**Additional context**
This appears to be a regression in 2.9, related to
https://github.com/apache/pulsar/pull/12192. I believe that two use cases are
broken:
1. Bookies are using a different ZK backend from that of the brokers.
2. The brokers are configured to use a chroot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
