michaeljmarshall opened a new pull request, #15818: URL: https://github.com/apache/pulsar/pull/15818
### Motivation The Pulsar broker, proxy, websocket proxy, and function worker each start several clients, including the Pulsar Client, the Pulsar Admin Client, and the Bookkeeper Client. Until now, there hasn't been an easy way to pass arbitrary configuration to these clients, which can cause problems. A recent example is the client memory limit that we fixed here: https://github.com/apache/pulsar/pull/15752. I propose that we enable pass through configuration for these clients, so that users can override any default configuration. Enable pass through configuration support for the bookkeeper clients created by `DistributedLog` and `BookKeeperPackagesStorage`. This support is already present for the managed ledger bookkeeper client, which takes all `bookkeeper_` prefixed configs. In order to simplify configuration, I reused the `bookkeeper_` prefix since it's reasonable to assume that these configs will be the same. Enable pass through configuration support for the broker clients using the `brokerClient_` prefix. Note that setting this once affects all clients. I think this is ideal, but it could lead to some confusion. A good secondary feature might be to add another prefix that targets specific clients, like the replication client. Initially implemented as https://github.com/apache/pulsar/pull/15763, but instead of submitting many PRs, I am just submitting one larger one. ### Modifications * Update all Broker, Proxy, WebSocket Proxy, and Function Worker clients so they can take arbitrary config. The Pulsar Client and the Pulsar Admin Client take configuration starting with `brokerClient_`. The Bookkeeper Client and the DLog Bookeeper Client take configuration starting with `bookkeeper_`. * Create `PropertiesUtils` class to reduce code duplication for mapping prefixed properties to the `Map<String, Object>` * Add `PulsarAdminBuilder#loadConf` method to the interface to match the same method in the Pulsar Client builder. * Update the `BookKeeperPackagesStorageConfiguration` to expose the underlying `properties` object. * Add documentation to the `broker.conf`, the `proxy.conf`, the `websocket.conf` and the `functions_worker.yml` files. ### Verifying this change I added several tests and also manually verified that the configuration overrides applied. ### Alternative Implementation I had initially wanted to run the override logic at the end of all other configuration. However, it doesn't work correctly because of the issue highlighted here https://github.com/apache/pulsar/issues/8509. It might be helpful to change the implementation of the `loadConf` logic so that it doesn't serialize secrets as `****`. ### Does this pull request potentially affect one of the following parts: This is a backwards compatible change. The only nuance is that the package management dlog bookie client will inherit the same configs that the broker already uses for the bookie client. I think this is preferable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
