Thanks, Magesh. All, I've made a few very minor changes to some JavaDocs and the signatures of the name-value pair methods in TopicSettings interface. I also described as a fifth rejected alternative why this KIP does not modify any topic settings for existing topics. All of these are pretty minor, but I'm happy to hear about issues or suggestions.
Since the above changes were very minor, I'll kick off a vote to accept this KIP unless I hear something in the next day or two. Note that this KIP has been around in virtually the exact form for over a year. Best regards, Randall On Wed, Aug 29, 2018 at 9:17 PM Magesh Nandakumar <mage...@confluent.io> wrote: > Randall, > > I originally thought that this proposal was a config only topic settings > and hence made the comment about configs being pass through. I just > realized that the connectors can also override and provide the > TopicSettings. With that in mind, I think the proposal looks great. > Looking forward to the feature. > > Thanks, > Magesh > > On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar <mage...@confluent.io> > wrote: > > > I was wondering if it would be much simpler to just do a pass-through so > > that we can support any topic setting added in Kafka without any code > > changes in connect. Since these are for topics that will have the actual > > data stream, users might possibly need more flexibility in terms of how > the > > topics get created. > > > > Thanks > > Magesh > > > > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rha...@gmail.com> wrote: > > > >> Do you think we should support name-value pairs, too? > >> > >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <mage...@confluent.io > > > >> wrote: > >> > >> > Randall, > >> > > >> > Thanks a lot for the KIP. I think this would be a great addition for > >> many > >> > source connectors. > >> > One clarification I had was regarding the topic settings that can be > >> > configured. Is it limited to the setting exposed in the TopicSettings > >> > interface? > >> > > >> > Thanks > >> > Magesh > >> > > >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rha...@gmail.com> > wrote: > >> > > >> > > Okay, after much delay let's try this again for AK 2.1. Has anyone > >> found > >> > > any concerns? Stephane suggested that we allow updating topic > >> > > configurations (everything but partition count). I'm unconvinced > that > >> > it's > >> > > worth the additional complexity in the implementation and the > >> > documentation > >> > > to explain the behavior. Changing several of the topic-specific > >> > > configurations have significant impact on broker behavior / > >> > functionality, > >> > > so IMO we need to proceed more cautiously. > >> > > > >> > > Stephane, do you have a particular use case in mind for updating > topic > >> > > configurations on an existing topic? > >> > > > >> > > Randall > >> > > > >> > > > >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> > >> wrote: > >> > > > >> > > > The KIP deadline for 1.1 has already passed, but I'd like to > restart > >> > this > >> > > > discussion so that we make the next release. I've not yet > addressed > >> the > >> > > > previous comment about *existing* topics, but I'll try to do that > >> over > >> > > the > >> > > > next few weeks. Any other comments/suggestions/questions? > >> > > > > >> > > > Best regards, > >> > > > > >> > > > Randall > >> > > > > >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> > >> > wrote: > >> > > > > >> > > >> Oops. Yes, I meant “replication factor”. > >> > > >> > >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> > wrote: > >> > > >> > > >> > > >> > Randall: > >> > > >> > bq. AdminClient currently allows changing the replication > >> factory. > >> > > >> > > >> > > >> > By 'replication factory' did you mean 'replication factor' ? > >> > > >> > > >> > > >> > Cheers > >> > > >> > > >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch < > rha...@gmail.com > >> > > >> > > >> wrote: > >> > > >> >> > >> > > >> >> Currently the KIP's scope is only topics that don't yet exist, > >> and > >> > we > >> > > >> have > >> > > >> >> to cognizant of race conditions between tasks with the same > >> > > connector. > >> > > >> I > >> > > >> >> think it is worthwhile to consider whether the KIP's scope > >> should > >> > > >> expand to > >> > > >> >> also address *existing* partitions, though it may not be > >> > appropriate > >> > > to > >> > > >> >> have as much control when changing the topic settings for an > >> > existing > >> > > >> >> topic. For example, changing the number of partitions (which > the > >> > KIP > >> > > >> >> considers a "topic-specific setting" even though technically > it > >> is > >> > > not) > >> > > >> >> shouldn't be done blindly due to the partitioning impacts, and > >> IIRC > >> > > you > >> > > >> >> can't reduce them (which we could verify before applying). > >> Also, I > >> > > >> don't > >> > > >> >> think the AdminClient currently allows changing the > replication > >> > > >> factory. I > >> > > >> >> think changing the topic configs is less problematic both from > >> what > >> > > >> makes > >> > > >> >> sense for connectors to verify/change and from what the > >> AdminClient > >> > > >> >> supports. > >> > > >> >> > >> > > >> >> Even if we decide that it's not appropriate to change the > >> settings > >> > on > >> > > >> an > >> > > >> >> existing topic, I do think it's advantageous to at least > notify > >> the > >> > > >> >> connector (or task) prior to the first record sent to a given > >> topic > >> > > so > >> > > >> that > >> > > >> >> the connector can fail or issue a warning if it doesn't meet > its > >> > > >> >> requirements. > >> > > >> >> > >> > > >> >> Best regards, > >> > > >> >> > >> > > >> >> Randall > >> > > >> >> > >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > >> > > >> >> steph...@simplemachines.com.au> wrote: > >> > > >> >> > >> > > >> >>> Hi Randall, > >> > > >> >>> > >> > > >> >>> Thanks for the KIP. I like it > >> > > >> >>> What happens when the target topic is already created but the > >> > > configs > >> > > >> do > >> > > >> >>> not match? > >> > > >> >>> i.e. wrong RF, num partitions, or missing / additional > configs? > >> > Will > >> > > >> you > >> > > >> >>> attempt to apply the necessary changes or throw an error? > >> > > >> >>> > >> > > >> >>> Thanks! > >> > > >> >>> Stephane > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > >> > > mathieu.fenn...@replicon.com > >> > > >> > > >> > > >> >>> wrote: > >> > > >> >>> > >> > > >> >>> Ah, yes, I see you a highlighted part that should've made > >> this > >> > > >> clear > >> > > >> >>> to me the first read. :-) Much clearer now! > >> > > >> >>> > >> > > >> >>> By the way, enjoyed your Debezium talk in NYC. > >> > > >> >>> > >> > > >> >>> Looking forward to this Kafka Connect change; it will > allow > >> me > >> > to > >> > > >> >>> remove a post-deployment tool that I hacked together for > the > >> > > >> purpose > >> > > >> >>> of ensuring auto-created topics have the right config. > >> > > >> >>> > >> > > >> >>> Mathieu > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch < > >> > > rha...@gmail.com> > >> > > >> >>> wrote: > >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first > >> > > >> >> configuration > >> > > >> >>> rule > >> > > >> >>>> whose regex matches will be applied, and no other rules will > >> be > >> > > >> >>> used. I've > >> > > >> >>>> updated the KIP to try to make this more clear, but let me > >> know > >> > if > >> > > >> >>> it's > >> > > >> >>>> still not clear. > >> > > >> >>>> > >> > > >> >>>> Best regards, > >> > > >> >>>> > >> > > >> >>>> Randall > >> > > >> >>>> > >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > >> > > >> >>>> mathieu.fenn...@replicon.com> wrote: > >> > > >> >>>> > >> > > >> >>>>> Hi Randall, > >> > > >> >>>>> > >> > > >> >>>>> Awesome, very much looking forward to this. > >> > > >> >>>>> > >> > > >> >>>>> It isn't 100% clear from the KIP how multiple config-based > >> rules > >> > > >> >>> would > >> > > >> >>>>> be applied; it looks like the first configuration rule > whose > >> > regex > >> > > >> >>>>> matches the topic name will be used, and no other rules > will > >> be > >> > > >> >>>>> applied. Is that correct? (I wasn't sure if it might > >> cascade > >> > > >> >>>>> together multiple matching rules...) > >> > > >> >>>>> > >> > > >> >>>>> Looks great, > >> > > >> >>>>> > >> > > >> >>>>> Mathieu > >> > > >> >>>>> > >> > > >> >>>>> > >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch < > >> > rha...@gmail.com> > >> > > >> >>> wrote: > >> > > >> >>>>>> Hi, all. > >> > > >> >>>>>> > >> > > >> >>>>>> We recently added the ability for Kafka Connect to create > >> > > >> >>> *internal* > >> > > >> >>>>> topics > >> > > >> >>>>>> using the new AdminClient, but it still would be great if > >> Kafka > >> > > >> >>> Connect > >> > > >> >>>>>> could do this for new topics that result from source > >> connector > >> > > >> >>> records. > >> > > >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka > >> Connect > >> > > >> >>> should > >> > > >> >>>>> allow > >> > > >> >>>>>> source connectors to set topic-specific settings for new > >> > > >> >> topics". > >> > > >> >>>>>> > >> > > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> > > >> >>>>> set+topic-specific+settings+for+new+topics > >> > > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> > > >> >>>>> set+topic-specific+settings+for+new+topics>* > >> > > >> >>>>>> > >> > > >> >>>>>> Please take a look and provide feedback. Thanks! > >> > > >> >>>>>> > >> > > >> >>>>>> Best regards, > >> > > >> >>>>>> > >> > > >> >>>>>> Randall > >> > > >> >>>>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >>> > >> > > >> >> > >> > > >> > >> > > > > >> > > > > >> > > > >> > > >> > > >
