Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Ryanne, I agree that worker property is named poorly. Maybe "disable.topic.creation.for.connectors"? I've updated the KIP to use this and to change the default to be enabled, but suggestions on better names are welcome. Ewen, you mentioned this property on the vote thread: > My only final

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Thanks Randall, makes sense to me. I suggest we change the topic.creation.enabled property name though. Sounds like it means topics are not created at all when disabled. Ryanne On Sat, Jan 19, 2019, 1:13 PM Randall Hauch Hi, > > Thanks again for all of the feedback. Based upon this feedback, I

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi, Thanks again for all of the feedback. Based upon this feedback, I do agree that we should indeed just solve the simple problem for the vast majority of use cases that only require a few simple properties. I believe I was the only person advocating for a more general and flexible solution, but

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

On Tue, Nov 27, 2018 at 6:31 PM Randall Hauch wrote: > Thanks for the feedback. Some thoughts inline. > > On Tue, Nov 27, 2018 at 5:47 PM Ewen Cheslack-Postava > wrote: > >> re: AdminClient vs this proposal, one consideration is that AdminClient >> exposes a lot more surface area and probably a

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, have you considered something like: - introduce TopicCreationPolicy interface, with methods like partitionsForTopic(topic). - provide a DefaultTopicCreationPolicy implementation that implements the current behavior. - provide a SimpleTopicCreationPolicy that honors

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Thanks for the feedback. Some thoughts inline. On Tue, Nov 27, 2018 at 5:47 PM Ewen Cheslack-Postava wrote: > re: AdminClient vs this proposal, one consideration is that AdminClient > exposes a lot more surface area and probably a bunch of stuff we actually > don't want Connectors to be able to

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

re: AdminClient vs this proposal, one consideration is that AdminClient exposes a lot more surface area and probably a bunch of stuff we actually don't want Connectors to be able to do, such as deleting topics. You can always lock down by ACLs, but what the framework enables directly vs requiring

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

FWIW, I’d find this feature useful. On Mon, Sep 24, 2018 at 2:42 PM Randall Hauch wrote: > Ryanne, > > If your connector is already using the AdminClient, then you as the > developer have a choice of switching to the new Connect-based functionality > or keeping the existing use of the

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Ryanne, If your connector is already using the AdminClient, then you as the developer have a choice of switching to the new Connect-based functionality or keeping the existing use of the AdminClient. If the connector uses both mechanisms (which I wouldn't recommend, simply because of the

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, Say I've got a connector that needs to control topic creation. What I need is an AdminClient s.t. my connector can do what it knows it needs to do. This KIP doesn't address the issues that have been brought up wrt configuration, principals, ACL etc, since I'll still need to construct my

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi, Ryanne. My apologies for not responding earlier, as I was on a long holiday. Thanks for your feedback and questions about this KIP. You've raised several points in the discussion so far, so let me try to address most of them. IIUC, one of your major concerns is that this KIP introduces a new

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

> Rather than go though the connect framework, connectors should just create their own AdminClient instance and create their own topics? Rather, can the framework be improved to expose an AdminClient ready to use? Then connectors can use this instance without needing separate

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

If an admin team gives another the right to create and configure their topic, I can't see why specifying topic configuration in connector properties wouldn't be a great idea. Configuration as code, repeatable, central and automated, easy iteration. Sign me in On Wed., 12 Sep. 2018, 4:57 am Gwen

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi Ryanne, Thanks for the feedback! Can you explain a bit more what you mean by "if we allow connectors to make this decision, they should have full control of the process."? I assume you mean, something like: Rather than go though the connect framework, connectors should just create their own

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, I have some concerns with this proposal. Firstly, I don't believe it is the job of a connector to configure topics, generally, nor for topic-specific settings to hang out in connector configurations. Automatic creation of topics with default settings is an established pattern elsewhere,

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Okay, I think I cleaned up the formatting issues in the KIP wiki page. And while implementing I realized that it'd be helpful to be able to limit via the connector configuration and the rules which topics are created. I added the `topic.creation.${ruleName}.policy` behavior, with possible values

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Thanks, Magesh. All, I've made a few very minor changes to some JavaDocs and the signatures of the name-value pair methods in TopicSettings interface. I also described as a fifth rejected alternative why this KIP does not modify any topic settings for existing topics. All of these are pretty

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, I originally thought that this proposal was a config only topic settings and hence made the comment about configs being pass through. I just realized that the connectors can also override and provide the TopicSettings. With that in mind, I think the proposal looks great. Looking forward

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

I was wondering if it would be much simpler to just do a pass-through so that we can support any topic setting added in Kafka without any code changes in connect. Since these are for topics that will have the actual data stream, users might possibly need more flexibility in terms of how the topics

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Do you think we should support name-value pairs, too? On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar wrote: > Randall, > > Thanks a lot for the KIP. I think this would be a great addition for many > source connectors. > One clarification I had was regarding the topic settings that can be >

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, Thanks a lot for the KIP. I think this would be a great addition for many source connectors. One clarification I had was regarding the topic settings that can be configured. Is it limited to the setting exposed in the TopicSettings interface? Thanks Magesh On Tue, Aug 21, 2018 at 7:59

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Okay, after much delay let's try this again for AK 2.1. Has anyone found any concerns? Stephane suggested that we allow updating topic configurations (everything but partition count). I'm unconvinced that it's worth the additional complexity in the implementation and the documentation to explain

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

The KIP deadline for 1.1 has already passed, but I'd like to restart this discussion so that we make the next release. I've not yet addressed the previous comment about *existing* topics, but I'll try to do that over the next few weeks. Any other comments/suggestions/questions? Best regards,

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Oops. Yes, I meant “replication factor”. > On Oct 4, 2017, at 7:18 PM, Ted Yu wrote: > > Randall: > bq. AdminClient currently allows changing the replication factory. > > By 'replication factory' did you mean 'replication factor' ? > > Cheers > >> On Wed, Oct 4, 2017 at

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall: bq. AdminClient currently allows changing the replication factory. By 'replication factory' did you mean 'replication factor' ? Cheers On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch wrote: > Currently the KIP's scope is only topics that don't yet exist, and we have >

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

I agree. I'm personally against increasing the partitions number, but RF would make sense. Same for configs, I'm okay with them being overriden. Maybe a "conflict" setting would make sense? Options: do nothing, throw exception, or apply? (default: do nothing - for safety) It'd be worth

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Currently the KIP's scope is only topics that don't yet exist, and we have to cognizant of race conditions between tasks with the same connector. I think it is worthwhile to consider whether the KIP's scope should expand to also address *existing* partitions, though it may not be appropriate to

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi Randall, Thanks for the KIP. I like it What happens when the target topic is already created but the configs do not match? i.e. wrong RF, num partitions, or missing / additional configs? Will you attempt to apply the necessary changes or throw an error? Thanks! Stephane On 24/5/17,

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Ah, yes, I see you a highlighted part that should've made this clear to me the first read. :-) Much clearer now! By the way, enjoyed your Debezium talk in NYC. Looking forward to this Kafka Connect change; it will allow me to remove a post-deployment tool that I hacked together for the purpose

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Thanks for the quick feedback, Mathieu. Yes, the first configuration rule whose regex matches will be applied, and no other rules will be used. I've updated the KIP to try to make this more clear, but let me know if it's still not clear. Best regards, Randall On Tue, May 23, 2017 at 10:07 AM,

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi Randall, Awesome, very much looking forward to this. It isn't 100% clear from the KIP how multiple config-based rules would be applied; it looks like the first configuration rule whose regex matches the topic name will be used, and no other rules will be applied. Is that correct? (I wasn't