Yes, the provider classes will need to be installed on each worker (same installation mechanism than a connector-plugin).
A new provider instance should be create for each connector instance but will be configure at the worker level. A provider class may have two behaviors : - provide a default configuration subset for connectors (a connector can still override these defaults). - enforce a subset of configuration (connectors can't override a provider configuration). This behavior could be configured at the worker level. Finally, I think a provider should be able to trigger a connector reconfiguration. 2017-10-23 17:30 GMT+02:00 Sönke Liebau <soenke.lie...@opencore.com.invalid> : > I agree, sounds like an intriguing idea. I think we could probably come up > with a few common enough implementations that merit including in Kafka. > FileConfigProvider for example, so you can distribute common configs > throughout your cluster with some orchestration tool and users simply state > the identifier of some connection.. > > What I am wondering is, whether these classes would always return an entire > configuration, or is it a more specific approach where you might use a > FileConfigProvider to retrieve some hostname and some other ConfigProvider > to retrieve credentials, etc... > > > > On Mon, Oct 23, 2017 at 5:12 PM, Randall Hauch <rha...@gmail.com> wrote: > > > Very interesting. Would the proposed configuration provider be set at the > > connector level or the worker level? The latter would obviously be > required > > to handle all/multiple connector configurations. Either way, the provider > > class(es) would need to be installed on the worker (really, every > worker), > > correct? > > > > Would all provider implementations be custom implementations, or are > there > > some provider implementations that are general enough for Connect to > > include them? > > > > Best regards, > > > > Randall > > > > On Fri, Oct 20, 2017 at 5:08 AM, Florian Hussonnois < > fhussonn...@gmail.com > > > > > wrote: > > > > > Hi Team > > > > > > Before submitting a new KIP I would like to open the discussion > regarding > > > an enhancement of Kafka Connect. > > > > > > Currently, the only way to configure a connector (in distributed mode) > is > > > through REST endpoints while creating or updating a connector. > > > > > > It would be nice to have the possibility to specify a configs provider > > > class (as we specify the connector class) in the JSON payload sent over > > the > > > REST API. > > > This class would be called during the connector creation to complete > the > > > configs submitted via REST. > > > > > > The motivations for a such functionality is for example to enforce a > > > configuration for all deployed connectors, to provide default configs > or > > to > > > provide sensitive configs like user/password. > > > > > > I've met these requirements on different projects. > > > > > > Do you think, this feature merits a new KIP ? > > > > > > Thanks, > > > > > > -- > > > Florian HUSSONNOIS > > > > > > > > > -- > Sönke Liebau > Partner > Tel. +49 179 7940878 > OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany > -- Florian HUSSONNOIS
