BewareMyPower commented on issue #24584: URL: https://github.com/apache/pulsar/issues/24584#issuecomment-3135204149
Regarding the performance bottlenecks, does it come from a test result of real workload? The metadata operations are not in the critical paths like produce or consume. In addition, there is a [metadata cache](https://github.com/apache/pulsar/blob/master/pulsar-metadata/src/main/java/org/apache/pulsar/metadata/cache/impl/MetadataCacheImpl.java) widely used internally. I admit that ZooKeeper has its limitation, that's why https://github.com/oxia-db/oxia is born. But implement a built-in metadata management system might not be a good idea. I can hardly see Pulsar users suffering ZooKeeper related issues in real production environments. This suggestion looks very similar to Kafka's KRaft proposal. However, it's different in Pulsar. Kafka has more heavy metadata. For example, a dynamically changed ISR list maintained for each topic. Before introducing KRaft, there is only one single centralized controller node in Kafka, which is not scalable. Well, now Kafka does not have a ZooKeeper cluster, but it has a KRaft controller cluster and its separated [KRaft Quorum Monitoring Metrics](https://kafka.apache.org/documentation/#kraft_quorum_monitoring). Each Pulsar broker can serve any metadata related request, and Pulsar brokers don't communicate with each other like Kafka brokers. That's the main difference. Actually, Pulsar can do the similar thing by renaming ZooKeeper as `pulsar-metadata` and `BookKeeper` as `pulsar-storage` so that it can claim that now Pulsar has only 1 component. But it's meaningless. IMHO, developing a better metadata service or even using a RDS might be good, but it's not a important part from a practical perspective. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
