Joforde commented on issue #23889: URL: https://github.com/apache/pulsar/issues/23889#issuecomment-2614188629
Thank you for your clear and accurate response; I found it very helpful. I will try the latest version of Oxia latter. In this case, I am using Pulsar version 4.0.2, with ZooKeeper as the metadata storage service. I have identified the critical issue: when a broker registers information with the metadata storage service, it sets the `expectedVersion` field to `Optional.empty()`. For implementations using ZooKeeper, this creates a PERSISTENT node, which means that the node cannot be destroyed when the broker stalls. - When the broker registers: https://github.com/apache/pulsar/blob/2a9d4ac85d8d786979afaa0b965cdb27375ae969/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java#L279 - During ZooKeeper data writing, since `p.getOptions()` is empty, it calls `setData`, leading to a NONODE error, and ultimately calls `internalStorePut` to create a PERSISTENT node: https://github.com/apache/pulsar/blob/2a9d4ac85d8d786979afaa0b965cdb27375ae969/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/impl/ZKMetadataStore.java#L339-L345 The verification method is as follows: ```java @Test public void zookeeperEphemeralKeys() throws Exception { final String key1 = newKey(); final String key2 = newKey(); @Cleanup MetadataStoreExtended store = MetadataStoreExtended.create(zks.getConnectionString(), MetadataStoreConfig.builder().build()); store.put(key1, "value-1".getBytes(), Optional.of(-1L), EnumSet.of(CreateOption.Ephemeral)).join(); store.put(key2, "value-1".getBytes(), Optional.empty(), EnumSet.of(CreateOption.Ephemeral)).join(); store.close(); @Cleanup MetadataStoreExtended store2 = MetadataStoreExtended.create(zks.getConnectionString(), MetadataStoreConfig.builder().build()); assertFalse(store2.exists(key1).join()); // This check will not pass assertFalse(store2.exists(key2).join()); store2.close(); } ``` > > However, if the broker is forced to exit due to issues like hardware failure, network problems, or being terminated with `kill -9`, it does not call the unregister method to delete its metadata. This results in the metadata for brokers that are no longer accessible remaining in the system. > > The metadata should get deleted automatically when the session expires. When a broker crashes, a new broker instance would not be able to start until the session expires and the empheral metadata entries expire. > > There's `EnumSet.of(CreateOption.Ephemeral)`: > > [pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java](https://github.com/apache/pulsar/blob/66d1bb0d734f12d758b0f0e9e3c0b42543508f8d/pulsar-broker/src/main/java/org/apache/pulsar/broker/loadbalance/extensions/BrokerRegistryImpl.java#L142) > > Line 142 in [66d1bb0](/apache/pulsar/commit/66d1bb0d734f12d758b0f0e9e3c0b42543508f8d) > > return brokerLookupDataMetadataCache.put(brokerIdKeyPath, brokerLookupData, EnumSet.of(CreateOption.Ephemeral)) > What metadata store implementation are you using? (What version, deployment type?) Just in case you are using Oxia since you are facing this issue. With Oxia there was a related bug, addressed by [streamnative/oxia#534](https://github.com/streamnative/oxia/pull/534) since Oxia v0.10.0. With Oxia, I believe it's worth tracking the [recent releases](https://github.com/streamnative/oxia/releases) and keeping up-to-date. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
