Re: Before creating KIP : Kafka Connect / Add a configuration provider class

[Florian Hussonnois]

Yes, the provider classes will need to be installed on each worker (same
installation mechanism than a connector-plugin).

A new provider instance should be create for each connector instance but
will be configure at the worker level.

A provider class may have two behaviors  :
 - provide a default configuration subset for connectors (a connector can
still override these defaults).
 - enforce a subset of configuration (connectors can't override a provider
configuration).

This behavior could be configured at the worker level.

Finally, I think a provider should be able to trigger a connector
reconfiguration.


2017-10-23 17:30 GMT+02:00 Sönke Liebau 
:

> I agree, sounds like an intriguing idea. I think we could probably come up
> with a few common enough implementations that merit including in Kafka.
> FileConfigProvider for example, so you can distribute common configs
> throughout your cluster with some orchestration tool and users simply state
> the identifier of some connection..
>
> What I am wondering is, whether these classes would always return an entire
> configuration, or is it a more specific approach where you might use a
> FileConfigProvider to retrieve some hostname and some other ConfigProvider
> to retrieve credentials, etc...
>
>
>
> On Mon, Oct 23, 2017 at 5:12 PM, Randall Hauch  wrote:
>
> > Very interesting. Would the proposed configuration provider be set at the
> > connector level or the worker level? The latter would obviously be
> required
> > to handle all/multiple connector configurations. Either way, the provider
> > class(es) would need to be installed on the worker (really, every
> worker),
> > correct?
> >
> > Would all provider implementations be custom implementations, or are
> there
> > some provider implementations that are general enough for Connect to
> > include them?
> >
> > Best regards,
> >
> > Randall
> >
> > On Fri, Oct 20, 2017 at 5:08 AM, Florian Hussonnois <
> fhussonn...@gmail.com
> > >
> > wrote:
> >
> > > Hi Team
> > >
> > > Before submitting a new KIP I would like to open the discussion
> regarding
> > > an enhancement of Kafka Connect.
> > >
> > > Currently, the only way to configure a connector (in distributed mode)
> is
> > > through REST endpoints while creating or updating a connector.
> > >
> > > It would be nice to have the possibility to specify a configs provider
> > > class (as we specify the connector class) in the JSON payload sent over
> > the
> > > REST API.
> > > This class would be called during the connector creation to complete
> the
> > > configs submitted via REST.
> > >
> > > The motivations for a such functionality is for example to enforce a
> > > configuration for all deployed connectors, to provide default configs
> or
> > to
> > > provide sensitive configs like user/password.
> > >
> > > I've met these requirements on different projects.
> > >
> > > Do you think, this feature merits a new KIP ?
> > >
> > > Thanks,
> > >
> > > --
> > > Florian HUSSONNOIS
> > >
> >
>
>
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany
>



-- 
Florian HUSSONNOIS

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

[Sönke Liebau]

I agree, sounds like an intriguing idea. I think we could probably come up
with a few common enough implementations that merit including in Kafka.
FileConfigProvider for example, so you can distribute common configs
throughout your cluster with some orchestration tool and users simply state
the identifier of some connection..

What I am wondering is, whether these classes would always return an entire
configuration, or is it a more specific approach where you might use a
FileConfigProvider to retrieve some hostname and some other ConfigProvider
to retrieve credentials, etc...



On Mon, Oct 23, 2017 at 5:12 PM, Randall Hauch  wrote:

> Very interesting. Would the proposed configuration provider be set at the
> connector level or the worker level? The latter would obviously be required
> to handle all/multiple connector configurations. Either way, the provider
> class(es) would need to be installed on the worker (really, every worker),
> correct?
>
> Would all provider implementations be custom implementations, or are there
> some provider implementations that are general enough for Connect to
> include them?
>
> Best regards,
>
> Randall
>
> On Fri, Oct 20, 2017 at 5:08 AM, Florian Hussonnois  >
> wrote:
>
> > Hi Team
> >
> > Before submitting a new KIP I would like to open the discussion regarding
> > an enhancement of Kafka Connect.
> >
> > Currently, the only way to configure a connector (in distributed mode) is
> > through REST endpoints while creating or updating a connector.
> >
> > It would be nice to have the possibility to specify a configs provider
> > class (as we specify the connector class) in the JSON payload sent over
> the
> > REST API.
> > This class would be called during the connector creation to complete the
> > configs submitted via REST.
> >
> > The motivations for a such functionality is for example to enforce a
> > configuration for all deployed connectors, to provide default configs or
> to
> > provide sensitive configs like user/password.
> >
> > I've met these requirements on different projects.
> >
> > Do you think, this feature merits a new KIP ?
> >
> > Thanks,
> >
> > --
> > Florian HUSSONNOIS
> >
>



-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

[Randall Hauch]

Very interesting. Would the proposed configuration provider be set at the
connector level or the worker level? The latter would obviously be required
to handle all/multiple connector configurations. Either way, the provider
class(es) would need to be installed on the worker (really, every worker),
correct?

Would all provider implementations be custom implementations, or are there
some provider implementations that are general enough for Connect to
include them?

Best regards,

Randall

On Fri, Oct 20, 2017 at 5:08 AM, Florian Hussonnois 
wrote:

> Hi Team
>
> Before submitting a new KIP I would like to open the discussion regarding
> an enhancement of Kafka Connect.
>
> Currently, the only way to configure a connector (in distributed mode) is
> through REST endpoints while creating or updating a connector.
>
> It would be nice to have the possibility to specify a configs provider
> class (as we specify the connector class) in the JSON payload sent over the
> REST API.
> This class would be called during the connector creation to complete the
> configs submitted via REST.
>
> The motivations for a such functionality is for example to enforce a
> configuration for all deployed connectors, to provide default configs or to
> provide sensitive configs like user/password.
>
> I've met these requirements on different projects.
>
> Do you think, this feature merits a new KIP ?
>
> Thanks,
>
> --
> Florian HUSSONNOIS
>

Before creating KIP : Kafka Connect / Add a configuration provider class

[Florian Hussonnois]

Hi Team

Before submitting a new KIP I would like to open the discussion regarding
an enhancement of Kafka Connect.

Currently, the only way to configure a connector (in distributed mode) is
through REST endpoints while creating or updating a connector.

It would be nice to have the possibility to specify a configs provider
class (as we specify the connector class) in the JSON payload sent over the
REST API.
This class would be called during the connector creation to complete the
configs submitted via REST.

The motivations for a such functionality is for example to enforce a
configuration for all deployed connectors, to provide default configs or to
provide sensitive configs like user/password.

I've met these requirements on different projects.

Do you think, this feature merits a new KIP ?

Thanks,

-- 
Florian HUSSONNOIS