Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2020-06-05 Thread Randall Hauch
LGTM. Thanks! On Fri, Jun 5, 2020 at 12:41 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Thanks for bringing up KIP-464 Jose and apologies for taking that long to > respond. > > It made sense to allow the users to use the broker defaults for the > replication factor and the

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2020-06-05 Thread Konstantine Karantasis
Thanks for bringing up KIP-464 Jose and apologies for taking that long to respond. It made sense to allow the users to use the broker defaults for the replication factor and the number of partitions when their source connectors create topics and the implementation has incorporated this ability.

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2020-02-03 Thread Jose Garcia Sancio
Thanks Konstantine. Looking forward to this feature. The KIP mentions: > For the *default* group this configuration is required. For any other group defined in topic.creation.groups this config is optional and if it's missing it gets the value the *default* group For the properties

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-19 Thread Tom Bentley
Thanks Konstantine, lgtm. On Thu, Dec 19, 2019 at 5:34 PM Ryanne Dolan wrote: > Thanks for the reply Konstantine. Makes sense. > > Ryanne > > On Tue, Dec 17, 2019, 6:41 PM Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > Thanks Randall and Ryanne for your comments. > > > > I'm

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-19 Thread Ryanne Dolan
Thanks for the reply Konstantine. Makes sense. Ryanne On Tue, Dec 17, 2019, 6:41 PM Konstantine Karantasis < konstant...@confluent.io> wrote: > Thanks Randall and Ryanne for your comments. > > I'm replying to them below, in order of appearance: > > To Randall's comments: > 1) I assumed these

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-18 Thread Konstantine Karantasis
Hi Tom, thanks for your comments, they are good points and useful corrections. Here's how things look now, after I updated the KIP: 10) I assumed proper escaping, but indeed it's better to be precise that every pattern will be treated as a regex. I also added an example with a topic name that

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-18 Thread Tom Bentley
Hi Konstantine, Thanks for the KIP. I can see this would be a useful addition to Kafka Connect. 10. The documentation of topic.creation.$alias.include says "A list of strings that represent either exact topic names or regular expressions that may match topic names." How do you know whether the

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-17 Thread Konstantine Karantasis
Thanks Randall and Ryanne for your comments. I'm replying to them below, in order of appearance: To Randall's comments: 1) I assumed these properties would be visible to connectors, since by definition these are connector properties. I added a mention. However I'm not sure if you are also making

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-12 Thread Ryanne Dolan
Konstantine, thanks for the updates. I wonder if we should take your proposal one step further and make this pluggable. Your include/exclude regexes are great out-of-the-box features, but it may be valuable to plug-in more sophisticated logic to handle topic creation. Instead of

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-12 Thread Randall Hauch
Thanks for updating and improving this KIP, Konstantine! Overall I'm pretty happy with the KIP, but I have a few fairly nit-picky questions/comments. 1) Will connectors see these configuration properties when Connect calls Connector.start(...)? We've added a number of configs to connectors that

[DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-12 Thread Konstantine Karantasis
I've taken a second look to KIP-158 after syncing with Randall Hauch, who was the original author of the proposal, and I have updated the KIP in place. The main new features of this updated KIP-158 is the introduction of groups of configs that can be composed and the ability to match topics to