savearray2 opened a new issue #8027:
URL: https://github.com/apache/pulsar/issues/8027
**Describe the bug**
I've set up a basic pulsar cluster and attempted to set bookie rack
information via ```pulsar-admin bookies set-bookie-rack```.
The following is the output from ```racks-placement```:
```
{
"default" : {
"koto-bk-aen-0" : {
"rack" : "aen/0"
},
"koto-bk-aen-1" : {
"rack" : "aen/1"
},
"koto-pl-aen-0" : {
"rack" : "aen/0"
},
"koto-pl-aen-1" : {
"rack" : "aen/1"
},
"koto-bk-aen0" : {
"rack" : "aen/0"
},
"koto-bk-aen1" : {
"rack" : "aen/1"
},
"koto-pl-aen0" : {
"rack" : "aen/0"
},
"koto-pl-aen1" : {
"rack" : "aen/1"
},
"koto-pulsar-aen0" : {
"rack" : "aen/0"
},
"koto-pulsar-aen1" : {
"rack" : "aen/1"
}
}
}
```
During the Pulsar broker client startup, I get the following error (please
note it says it can't find ```koto-bk-aen-0``` even though it's clearly
specified above):
```
07:41:34.078 [main] ERROR
org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to
initialize DNS Resolver
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping, used default subnet
resolver : java.lang.RuntimeException: java.util.concurrent.ExecutionException:
java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
07:41:34.079 [ForkJoinPool.commonPool-worker-1] WARN
com.github.benmanes.caffeine.cache.LocalAsyncLoadingCache - Exception thrown
during asynchronous load
java.lang.RuntimeException: java.net.UnknownHostException: koto-bk-aen-0
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:103)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.LinkedHashMap.forEach(LinkedHashMap.java:684)
~[?:1.8.0_252]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$0(ZkBookieRackAffinityMapping.java:85)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at java.util.TreeMap.forEach(TreeMap.java:1005) ~[?:1.8.0_252]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.updateRacksWithHost(ZkBookieRackAffinityMapping.java:84)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.access$3(ZkBookieRackAffinityMapping.java:78)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:154)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping$2.deserialize(ZkBookieRackAffinityMapping.java:1)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at
org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$13(ZooKeeperCache.java:362)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:598)
~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
~[org.apache.pulsar-pulsar-zookeeper-2.6.1.jar:2.6.1]
Caused by: java.net.UnknownHostException: koto-bk-aen-0
at
org.apache.bookkeeper.net.BookieSocketAddress.<init>(BookieSocketAddress.java:68)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.pulsar.zookeeper.ZkBookieRackAffinityMapping.lambda$1(ZkBookieRackAffinityMapping.java:87)
~[org.apache.pulsar-pulsar-zookeeper-utils-2.6.1.jar:2.6.1]
... 11 more
```
The bookies are then added to the default region/default rack, ignoring the
settings from ```racks-placement```:
```
07:41:34.139 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node:
/default-region/default-rack/koto-bk-aen-0:3181
07:41:34.139 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize
rackaware ensemble placement policy @ <Bookie:172.28.131.158:0> @
/default-region/default-rack :
org.apache.bookkeeper.client.TopologyAwareEnsemblePlacementPolicy$DNSResolverDecorator.
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node:
/default-region/default-rack/koto-bk-aen-1:3181
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node:
/default-region/default-rack/koto-bk-aen-0:3181
07:41:34.140 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO
org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node:
/default-region/default-rack/koto-bk-aen-1:3181
```
Relevant portions of the ```broker.conf``` file:
```
zookeeperServers=koto-aen-zk0:2181,koto-aen-zk1:2181,koto-aen-zk2:2181
configurationStoreServers=koto-cfg-zk0:2184,koto-cfg-zk1:2184,koto-cfg-zk2:2184,koto-cfg-aen-zk0:2184,koto-cfg-aen-zk1:2184,koto-cfg-aen-zk2:2184
bindAddress=0.0.0.0
advertisedAddress=koto-pulsar-aen0
bookkeeperClientRackawarePolicyEnabled=true
bookkeeperClientRegionawarePolicyEnabled=true
bookkeeperClientReorderReadSequenceEnabled=true
bookkeeperClientIsolationGroups=
bookkeeperClientSecondaryIsolationGroups=
bookkeeperClientMinAvailableBookiesInIsolationGroups=
```
**Expected behavior**
The bookies should be added to regions/racks based on the ```pulsar-admin```
settings.
**Version Info:**
- Using the official docker image: ```apachepulsar/pulsar-all:latest```
- Version: ```07:41:32.588 [main] INFO
org.apache.pulsar.broker.PulsarService - Starting Pulsar Broker service;
version: '2.6.1'```
**Additional context**
Not being able to properly specify region/rack settings really defeats the
purpose of Pulsar's scalability. Ensuring that Pulsar writes to bookies in
different regions is an important feature. Hopefully someone can help with
this...
I'd really, really like to get this working :)
Also, if necessary, I'm willing to clone the project and start digging
through the source, if contributions are needed/welcome (and this isn't being
taken care of elsewhere).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
