Demogorgon314 opened a new pull request, #21313: URL: https://github.com/apache/pulsar/pull/21313
(cherry picked from commit f85e0dc085d47587037cbc3f7d5443745384d350) ### Motivation Currently, if the cluster has multiple brokers, and the cluster is doing rolling restart, the heartbeat namespace topic's lookup result might be wrong, because the `ExtensibleLoadManagerImpl` does not check the heartbeat and SLA namespace bundle lookup candidate broker to let them own by the specified broker. This ownership selection is wrong: ``` 2023-09-20T08:27:32,188+0000 [ForkJoinPool.commonPool-worker-1] INFO org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl - Selected new owner broker: broker-2:8080 for bundle: pulsar/broker-0:8080/0x00000000_0xffffffff. ``` After the ownership assignment, the broker-0 will fail to start. ``` pulsar-broker 2023-09-20T08:27:54,707+0000 [main] INFO org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl - Try acquiring ownership for bundle: pulsar/broker-0:8080/0x00000000_0xffffffff - broker-0:8080. pulsar-broker 2023-09-20T08:27:54,707+0000 [main] ERROR org.apache.pulsar.broker.namespace.NamespaceService - namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker 2023-09-20T08:27:54,710+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:403) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker Caused by: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker ... 4 more pulsar-broker 2023-09-20T08:27:54,710+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service. pulsar-broker org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:938) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker Caused by: org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:403) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker ... 2 more pulsar-broker Caused by: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650 pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT] pulsar-broker ... 2 more pulsar-broker 2023-09-20T08:27:54,711+0000 [main] WARN org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1 ``` ### Modifications * Fix lookup heartbeat and sla namespace bundle when using extensible load manager * When updating the topK bundles, filter out the heartbeat and SLA namespace bundles. * Skip override orphan heartbeat namespace bundle ### Documentation <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. --> - [ ] `doc` <!-- Your PR contains doc changes. --> - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later --> - [x] `doc-not-needed` <!-- Your PR changes do not impact docs --> - [ ] `doc-complete` <!-- Docs have been already added --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
