poorbarcode commented on code in PR #23033: URL: https://github.com/apache/pulsar/pull/23033#discussion_r1687331377
########## pip/pip-366.md: ########## @@ -0,0 +1,70 @@ +# PIP-366: Support to specify different config for Configuration and Local Metadata Store + +# Background knowledge + +Pulsar metadata store maintains all the metadata, configuration, and coordination of a Pulsar cluster, such as topic metadata, schema, broker load data, and so on. + +The metadata store of each Pulsar instance should contain the following two components: + +- A local metadata store ensemble (`metadataStoreUrl`) that stores cluster-specific configuration and coordination, such as which brokers are responsible for which topics as well as ownership metadata, broker load reports, and BookKeeper ledger metadata. +- A configuration store quorum (`configurationMetadataStoreUrl`) stores configuration for clusters, tenants, namespaces, topics, and other entities that need to be globally consistent. + +# Motivation + +When using Geo-Replication and global configuration store for configuration global consistency, the configuration store's config may be different from the local metadata store's config. For example, the configuration store may have a different set of ZooKeeper servers than the local metadata store. + +The global configuration store may deploy in a different data center, and the local metadata store may be deployed in the same data center as the Pulsar broker. In this case, the global configuration store may need to use TLS and authentication to protect the connection to metadata store server, while the local metadata store may not need to use TLS and authentication. + +However, the current implementation of Pulsar only supports configuring different metadata store url for the local metadata store and the configuration store. This limitation makes it impossible to support the above scenario. + +# Goals + +## In Scope + +- Support specifying different configurations for the local metadata store and the configuration store. + +# Detailed Design + +## Design & Implementation Details + +Pulsar support `metadataStoreConfigPath` configuration, but it only supports for `RocksdbMetadataStore`, and it is not able to specify different configuration for Configuration Metadata Store. Review Comment: **background** - There is an interface called `MetadataStoreProvider` that is used to load up any metadata store. - Since `metadataStoreConfigPath` links to a file, we can write anything into the file so that it can be applied for all scenarios. Such as below: **file://metadataStoreConfig** ``` configMetadataStore_tlsTrustCertsFilePath=/tmp/a.key metadataStore_tlsTrustCertsFilePath=/tmp/b.key ``` So this PIP maybe can be discarded -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
