Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Colin, Thanks for the suggestion, I like it. Do others like the idea of a DirectoryConfigProvider, and if so is it ok if I amend the KIP? Thanks, Robert On Tue, Sep 4, 2018 at 5:33 PM, Colin McCabe wrote: > Hi Robert, > > This seems like a reasonable behavior to me. However, adding this >

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, This seems like a reasonable behavior to me. However, adding this behavior to FileConfigProvider seems like it might give someone who accidentally configures a directory rather than a file a nasty surprise. How about adding a DirectoryConfigProvider which adds this behavior? best,

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi everyone, Currently the FileConfigProvider, when passed a path that represents a Properties file, will read the file as a set of key-value pairs. I've filed https://issues.apache.org/jira/browse/KAFKA-7370, which proposes to augment FileConfigProvider so that when a path representing a directo

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Sounds great, happy to hear we all agree! Thanks everyone! - Konstantine On Mon, Jun 11, 2018 at 4:22 PM, Colin McCabe wrote: > Sounds good. Thanks, Konstantin. > > Colin > > > On Mon, Jun 11, 2018, at 13:41, Rajini Sivaram wrote: > > Hi Konstantine, > > > > Sounds reasonable to me too. > >

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Sounds good. Thanks, Konstantin. Colin On Mon, Jun 11, 2018, at 13:41, Rajini Sivaram wrote: > Hi Konstantine, > > Sounds reasonable to me too. > > Regards, > > Rajini > > On Mon, Jun 11, 2018 at 7:55 PM, Robert Yokota wrote: > > > Hi Konstantine, > > > > Sounds reasonable! > > > > Thanks

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Konstantine, Sounds reasonable to me too. Regards, Rajini On Mon, Jun 11, 2018 at 7:55 PM, Robert Yokota wrote: > Hi Konstantine, > > Sounds reasonable! > > Thanks, > Robert > > On Mon, Jun 11, 2018 at 11:49 AM, Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > Hi everyone,

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Konstantine, Sounds reasonable! Thanks, Robert On Mon, Jun 11, 2018 at 11:49 AM, Konstantine Karantasis < konstant...@confluent.io> wrote: > Hi everyone, after fixing an issue with a regular expression in Connect's > class loading isolation of the new component type ConfigProvider here: > >

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi everyone, after fixing an issue with a regular expression in Connect's class loading isolation of the new component type ConfigProvider here: https://github.com/apache/kafka/pull/5177 I noticed that the new interface ConfigProvider, along with its first implementation FileConfigProvider, have

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Thanks for the update, Robert. Looks good to me. Regards, Rajini On Wed, May 16, 2018 at 4:43 PM, Robert Yokota wrote: > Hi Rajini, > > Thanks for the excellent feedback! > > I've made the API changes that you've requested in the KIP. > > > > 1. Are we expecting one provider instance with diff

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Rajini, Thanks for the excellent feedback! I've made the API changes that you've requested in the KIP. > 1. Are we expecting one provider instance with different contexts > provided to `ConfigProvider.get()`? If we created a different provider > instance for each context, we could deal with

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Thanks for the KIP updates. The interfaces look suitable for brokers, with some small changes. If we can adapt the interface to implement the existing DynamicBrokerConfig, then we are good. With broker configs: 1. We don't know what configs are in ZK since we allow custom configs.

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Thanks, Ron! I will take a look. Regards, Robert On Tue, May 15, 2018 at 5:59 PM, Ron Dagostino wrote: > Hi Robert. Regarding your comment "use the lease duration to schedule a > configuration reload in the future", you might be interested in the code > that does refresh for OAuth Bearer Toke

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert. Regarding your comment "use the lease duration to schedule a configuration reload in the future", you might be interested in the code that does refresh for OAuth Bearer Tokens in KIP-255; specifically, the class org.apache.kafka.common.security.oauthbearer.internal.expiring.ExpiringCred

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Colin, Thanks for the feedback! > The KIP says that "Vault is very popular and has been described as 'the current gold standard in secret management and provisioning'." I think this might be a bit too much detail -- we don't really need to > favorites, right? :) I've removed this line :)

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Thanks for posting this. In the past we've been kind of reluctant to add more complexity to configuration. I think Connect does have a clear need for this kind of functionality, though. As you mention, Connect integrates with external systems, which are very likely to have passwor

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Magesh, I updated the KIP with a link to a PR for a working prototype. The prototype does not yet use the Connect plugin machinery for class loader isolation, but should give you an idea of what the final implementation will look like. Here is the link: https://github.com/apache/kafka/pull/49

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Ron, Thanks for the pointer! I'll take a look at KIP 269. I will also put together a PR to demonstrate what the implementation might look like. Thanks, Robert On Wed, May 9, 2018 at 6:44 AM, Ron Dagostino wrote: > Hi Robert. You may find KIP-269: Substitution Within Configuration Values

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Magesh, Thanks for the feedback! I will put together a PR to demonstrate what the implementation might look like, as well as a reference FileConfigProvider. 1. The delayMs for a (potentially) scheduled reload is determined by the ConfigProvider. For example, a (hypothetical) VaultConfigProv

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert. You may find KIP-269: Substitution Within Configuration Values ( https://cwiki.apache.org/confluence/display/KAFKA/KIP+ 269+Substitution+Within+Configuration+Values) to be interesting. I don't know if it is relevant here -- I agree with Magesh, more concrete details would help, includi

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Thanks for the KIP. I think, this will be a great addition to the framework. I think, will be great if the KIP can elaborate a little bit more on how implementations would look like with an example. Also, would be good to provide a reference implementation as well. The other questions

[DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi, I would like to start a discussion for KIP-297 to externalize secrets from Kafka Connect configurations. Any feedback is appreciated. < https://cwiki.apache.org/confluence/display/KAFKA/KIP-297%3A+Externalizing+Secrets+for+Connect+Configurations > JIRA: