yangl opened a new pull request, #15288:
URL: https://github.com/apache/pulsar/pull/15288
### Motivation
when we use the `metadataServiceUri` instead of the `zkServers` &
`zkLedgersRootPath` in the bookkeeper.conf and enable the rack-aware, the
BookieRackAffinityMapping doesn't work well, config as follow :
```xml
metadataServiceUri=zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers
#
zkServers=10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181
# zkLedgersRootPath=/ledgers
```
**the root cause was the we doesn't process the prefix `zk+hierarchical://`
or `zk+null` logic in the `BookieRackAffinityMapping`.**
```
2022-04-24 11:22:25,023 INFO [Main#117] [main] Using configuration file
conf/bookkeeper1.conf
2022-04-24 11:22:25,040 INFO [Main#265] [main] Hello, I'm your bookie,
bookieId is <not-set>, listening on port 3181. Metadata service uri is
zk+null://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers.
Journals are in [/tmp/bk-journal-1]. Ledgers are stored in
[data/bookkeeper/ledgers].
2022-04-24 11:22:25,059 INFO [Main#301] [main] Load lifecycle component :
org.apache.bookkeeper.server.service.StatsProviderService
2022-04-24 11:22:25,331 INFO [BookieServer#111] [main] {
"readBufferSizeBytes" : "4096",
"statsProviderClass" :
"org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider",
"majorCompactionThreshold" : "0.5",
"numJournalCallbackThreads" : "8",
"httpServerPort" : "8000",
"lostBookieRecoveryDelay" : "0",
"journalAlignmentSize" : "4096",
"compactionRateByBytes" : "1000000",
"httpServerClass" : "org.apache.bookkeeper.http.vertx.VertxHttpServer",
"minNumRacksPerWriteQuorum" : "2",
"dbStorage_rocksDB_numFilesInLevel0" : "4",
"minUsableSizeForIndexFileCreation" : "1073741824",
"gcOverreplicatedLedgerWaitTime" : "86400000",
"journalMaxGroupWaitMSec" : "1",
"dbStorage_rocksDB_numLevels" : "-1",
"dbStorage_rocksDB_bloomFilterBitsPerKey" : "10",
"nettyMaxFrameSizeBytes" : "5253120",
"readWorkerThreadsThrottlingEnabled" : "true",
"ledgerStorageClass" :
"org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage",
"auditorPeriodicBookieCheckInterval" : "86400",
"gcWaitTime" : "900000",
"compactionRate" : "1000",
"fileInfoFormatVersionToWrite" : "0",
"entryLogFilePreallocationEnabled" : "true",
"//metadataServiceUri" :
"zk+hierarchical://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers",
"journalSyncData" : "true",
"compactionRateByEntries" : "1000",
"dbStorage_rocksDB_maxSizeInLevel1MB" : "256",
"diskCheckInterval" : "10000",
"auditorPeriodicCheckInterval" : "604800",
"dbStorage_rocksDB_writeBufferSizeMB" : "64",
"autoRecoveryDaemonEnabled" : "true",
"maxPendingAddRequestsPerThread" : "10000",
"flushEntrylogBytes" : "268435456",
"majorCompactionInterval" : "86400",
"httpServerEnabled" : "false",
"flushInterval" : "60000",
"enforceMinNumRacksPerWriteQuorum" : "false",
"journalFlushWhenQueueEmpty" : "false",
"minorCompactionInterval" : "3600",
"dbStorage_rocksDB_blockCacheSize" : "",
"isThrottleByBytes" : "false",
"ensemblePlacementPolicy" :
"org.apache.pulsar.bookie.rackawareness.IsolatedBookieEnsemblePlacementPolicy",
"numAddWorkerThreads" : "0",
"dbStorage_rocksDB_sstSizeInMB" : "64",
"journalWriteBufferSizeKB" : "64",
"diskUsageThreshold" : "0.95",
"openFileLimit" : "0",
"prometheusStatsHttpPort" : "8000",
"enableBusyWait" : "false",
"journalMaxSizeMB" : "2048",
"journalAdaptiveGroupWrites" : "true",
"openLedgerRereplicationGracePeriod" : "30000",
"ledgerDirectories" : "data/bookkeeper/ledgers",
"zkTimeout" : "30000",
"dbStorage_rocksDB_blockSize" : "65536",
"journalMaxBackups" : "5",
"metadataServiceUri" :
"zk+null://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers",
"useV2WireProtocol" : "true",
"journalDirectories" : "/tmp/bk-journal-1",
"maxPendingReadRequestsPerThread" : "2500",
"useHostNameAsBookieID" : "true",
"rereplicationEntryBatchSize" : "100",
"allowLoopback" : "false",
"readOnlyModeEnabled" : "true",
"reppDnsResolverClass" :
"org.apache.pulsar.bookie.rackawareness.BookieRackAffinityMapping",
"journalRemoveFromPageCache" : "true",
"dbStorage_readAheadCacheMaxSizeMb" : "",
"zkEnableSecurity" : "false",
"numHighPriorityWorkerThreads" : "8",
"dbStorage_readAheadCacheBatchSize" : "1000",
"journalFormatVersionToWrite" : "5",
"writeBufferSizeBytes" : "65536",
"bookiePort" : "3181",
"dbStorage_writeCacheMaxSizeMb" : "",
"pageLimit" : "0",
"logSizeLimit" : "1073741824",
"advertisedAddress" : "",
"bookieDeathWatchInterval" : "1000",
"numReadWorkerThreads" : "8",
"minorCompactionThreshold" : "0.2",
"serverTcpNoDelay" : "true",
"journalBufferedWritesThreshold" : "524288",
"compactionMaxOutstandingRequests" : "100000",
"journalPreAllocSizeMB" : "16",
"journalWriteData" : "false"
}
2022-04-24 11:22:25,947 INFO [BookieNettyServer#357] [main] Binding
bookie-rpc endpoint to 0.0.0.0/0.0.0.0:3181
2022-04-24 11:22:26,256 INFO [MetadataDrivers#105] [main] BookKeeper
metadata driver manager initialized
2022-04-24 11:22:26,264 INFO [ZKMetadataDriverBase#197] [main] Initialize
zookeeper metadata driver at metadata service uri
zk+null://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers
: zkServers =
10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181,
ledgersRootPath = /ledgers.
2022-04-24 11:22:26,269 INFO [Environment#98] [main] Client
environment:zookeeper.version=3.8.0-5a02a05eddb59aee6ac762f7ea82e92a68eb9c0f,
built on 2022-02-25 08:49 UTC
2022-04-24 11:22:26,269 INFO [Environment#98] [main] Client
environment:java.version=17.0.2
2022-04-24 11:22:26,269 INFO [Environment#98] [main] Client
environment:java.vendor=BellSoft
2022-04-24 11:22:26,269 INFO [Environment#98] [main] Client
environment:java.home=D:\JAVA\jdk-17-full
2022-04-24 11:22:26,269 INFO [Environment#98] [main] Client
environment:java.class.path=
2022-04-24 11:22:26,270 INFO [Environment#98] [main] Client
environment:java.compiler=<NA>
2022-04-24 11:22:26,270 INFO [Environment#98] [main] Client
environment:os.name=Windows 7
2022-04-24 11:22:26,270 INFO [Environment#98] [main] Client
environment:os.arch=amd64
2022-04-24 11:22:26,271 INFO [Environment#98] [main] Client
environment:os.version=6.1
2022-04-24 11:22:26,271 INFO [Environment#98] [main] Client
environment:user.dir=D:\opensource\pulsar
2022-04-24 11:22:26,271 INFO [Environment#98] [main] Client
environment:os.memory.free=204MB
2022-04-24 11:22:26,271 INFO [Environment#98] [main] Client
environment:os.memory.max=4074MB
2022-04-24 11:22:26,271 INFO [Environment#98] [main] Client
environment:os.memory.total=256MB
2022-04-24 11:22:26,276 INFO [ZooKeeper#637] [main] Initiating client
connection,
connectString=10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181
sessionTimeout=30000
watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@7e8e8651
2022-04-24 11:22:26,280 INFO [X509Util#77] [main] Setting -D
jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS
renegotiation
2022-04-24 11:22:26,285 INFO [ClientCnxnSocket#239] [main] jute.maxbuffer
value is 1048575 Bytes
2022-04-24 11:22:26,294 INFO [ClientCnxn#1732] [main]
zookeeper.request.timeout value is 0. feature enabled=false
2022-04-24 11:22:26,304 INFO [ClientCnxn$SendThread#1171]
[main-SendThread(10.206.128.157:2181)] Opening socket connection to server
bookie34/10.206.128.157:2181.
2022-04-24 11:22:26,305 INFO [ClientCnxn$SendThread#1173]
[main-SendThread(10.206.128.157:2181)] SASL config status: Will not attempt to
authenticate using SASL (unknown error)
2022-04-24 11:22:26,310 INFO [ClientCnxn$SendThread#1005]
[main-SendThread(10.206.128.157:2181)] Socket connection established,
initiating session, client: /192.168.255.10:60376, server:
bookie34/10.206.128.157:2181
2022-04-24 11:22:26,904 INFO [ClientCnxn$SendThread#1444]
[main-SendThread(10.206.128.157:2181)] Session establishment complete on server
bookie34/10.206.128.157:2181, session id = 0x4000000f2840009, negotiated
timeout = 30000
2022-04-24 11:22:26,908 INFO [ZooKeeperWatcherBase#130] [main-EventThread]
ZooKeeper client is connected now.
2022-04-24 11:22:27,136 INFO [Bookie#413] [main] Stamping new cookies on
all dirs [\tmp\bk-journal-1\current] [data\bookkeeper\ledgers\current]
2022-04-24 11:22:27,215 INFO [Bookie#736] [main] instantiate ledger manager
org.apache.bookkeeper.meta.HierarchicalLedgerManagerFactory
2022-04-24 11:22:27,256 INFO [Bookie#654] [main] Using ledger storage:
org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage
2022-04-24 11:22:27,257 INFO [DbLedgerStorage#108] [main] Started Db Ledger
Storage
2022-04-24 11:22:27,258 INFO [DbLedgerStorage#109] [main] - Number of
directories: 1
2022-04-24 11:22:27,258 INFO [DbLedgerStorage#110] [main] - Write cache
size: 1018 MB
2022-04-24 11:22:27,258 INFO [DbLedgerStorage#111] [main] - Read Cache:
1018 MB
2022-04-24 11:22:27,262 INFO [SingleDirectoryDbLedgerStorage#149] [main]
Creating single directory db ledger storage on data\bookkeeper\ledgers\current
2022-04-24 11:22:34,976 INFO [ScanAndCompareGarbageCollector#102] [main]
Over Replicated Ledger Deletion : enabled=true, interval=86400000
2022-04-24 11:22:35,024 INFO [GarbageCollectorThread#245] [main] Minor
Compaction : enabled=true, threshold=0.2, interval=3600000
2022-04-24 11:22:35,024 INFO [GarbageCollectorThread#247] [main] Major
Compaction : enabled=true, threshold=0.5, interval=86400000
2022-04-24 11:22:35,095 INFO [Main#308] [main] Load lifecycle component :
org.apache.bookkeeper.server.service.BookieService
2022-04-24 11:22:35,202 INFO [ZKMetadataDriverBase#197] [main] Initialize
zookeeper metadata driver at metadata service uri
zk+null://10.206.128.154:2181;10.206.128.155:2181;10.206.128.156:2181;10.206.128.157:2181;10.206.128.158:2181/ledgers
: zkServers =
10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181,
ledgersRootPath = /ledgers.
2022-04-24 11:22:35,203 INFO [ZooKeeper#637] [main] Initiating client
connection,
connectString=10.206.128.154:2181,10.206.128.155:2181,10.206.128.156:2181,10.206.128.157:2181,10.206.128.158:2181
sessionTimeout=30000
watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@58112bc4
2022-04-24 11:22:35,208 INFO [ClientCnxnSocket#239] [main] jute.maxbuffer
value is 1048575 Bytes
2022-04-24 11:22:35,209 INFO [ClientCnxn#1732] [main]
zookeeper.request.timeout value is 0. feature enabled=false
2022-04-24 11:22:35,215 INFO [ClientCnxn$SendThread#1171]
[main-SendThread(10.206.128.156:2181)] Opening socket connection to server
bookie33/10.206.128.156:2181.
2022-04-24 11:22:35,215 INFO [ClientCnxn$SendThread#1173]
[main-SendThread(10.206.128.156:2181)] SASL config status: Will not attempt to
authenticate using SASL (unknown error)
2022-04-24 11:22:35,219 INFO [ClientCnxn$SendThread#1005]
[main-SendThread(10.206.128.156:2181)] Socket connection established,
initiating session, client: /192.168.255.10:60386, server:
bookie33/10.206.128.156:2181
2022-04-24 11:22:35,816 INFO [ClientCnxn$SendThread#1444]
[main-SendThread(10.206.128.156:2181)] Session establishment complete on server
bookie33/10.206.128.156:2181, session id = 0x303a43aba0002da, negotiated
timeout = 30000
2022-04-24 11:22:35,817 INFO [ZooKeeperWatcherBase#130] [main-EventThread]
ZooKeeper client is connected now.
2022-04-24 11:22:43,495 ERROR [Main#228] [main] Failed to build bookie server
java.lang.RuntimeException: METADATA_STORE_INSTANCE failed initialized
at
org.apache.pulsar.bookie.rackawareness.IsolatedBookieEnsemblePlacementPolicy.initialize(IsolatedBookieEnsemblePlacementPolicy.java:78)
~[classes/:?]
at
org.apache.pulsar.bookie.rackawareness.IsolatedBookieEnsemblePlacementPolicy.initialize(IsolatedBookieEnsemblePlacementPolicy.java:52)
~[classes/:?]
at
org.apache.bookkeeper.client.BookKeeper.initializeEnsemblePlacementPolicy(BookKeeper.java:581)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at org.apache.bookkeeper.client.BookKeeper.<init>(BookKeeper.java:505)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.client.BookKeeper$Builder.build(BookKeeper.java:306)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.replication.Auditor.createBookKeeperClient(Auditor.java:280)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.replication.AutoRecoveryMain.<init>(AutoRecoveryMain.java:95)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at
org.apache.bookkeeper.server.service.AutoRecoveryService.<init>(AutoRecoveryService.java:41)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at org.apache.bookkeeper.server.Main.buildBookieServer(Main.java:320)
~[bookkeeper-server-4.14.4.jar:4.14.4]
at org.apache.bookkeeper.server.Main.doMain(Main.java:226)
[bookkeeper-server-4.14.4.jar:4.14.4]
at org.apache.bookkeeper.server.Main.main(Main.java:208)
[bookkeeper-server-4.14.4.jar:4.14.4]
at org.apache.bookkeeper.proto.BookieServer.main(BookieServer.java:334)
[bookkeeper-server-4.14.4.jar:4.14.4]
Disconnected from the target VM, address: '127.0.0.1:60340', transport:
'socket'
```
### Modifications
1. delete the `zk+hierarchical://` or `zk+null://` prefix
2. move the `TestZKServer` to the `testmocks` module
3. add the testcase
https://github.com/apache/pulsar/blob/0a38fce64f9caba68eb9d7e6828f9833815f5adc/pulsar-broker-common/src/main/java/org/apache/pulsar/bookie/rackawareness/BookieRackAffinityMapping.java#L85-L86
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
*(Please pick either of the following options)*
This change is a trivial rework / code cleanup without any test coverage.
*(or)*
This change is already covered by existing tests, such as *(please describe
tests)*.
*(or)*
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end deployment with large payloads
(10MB)*
- *Extended integration test for recovery after broker failure*
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (yes / no)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
- The rest endpoints: (yes / no)
- The admin cli options: (yes / no)
- Anything that affects deployment: (yes / no / don't know)
### Documentation
Check the box below or label this PR directly.
Need to update docs?
- [ ] `doc-required`
(Your PR needs to update docs and you will update later)
- [x] `no-need-doc`
(Please explain why)
- [ ] `doc`
(Your PR contains doc changes)
- [ ] `doc-added`
(Docs have been already added)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
