Randall, I originally thought that this proposal was a config only topic settings and hence made the comment about configs being pass through. I just realized that the connectors can also override and provide the TopicSettings. With that in mind, I think the proposal looks great. Looking forward to the feature.
Thanks, Magesh On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar <mage...@confluent.io> wrote: > I was wondering if it would be much simpler to just do a pass-through so > that we can support any topic setting added in Kafka without any code > changes in connect. Since these are for topics that will have the actual > data stream, users might possibly need more flexibility in terms of how the > topics get created. > > Thanks > Magesh > > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rha...@gmail.com> wrote: > >> Do you think we should support name-value pairs, too? >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <mage...@confluent.io> >> wrote: >> >> > Randall, >> > >> > Thanks a lot for the KIP. I think this would be a great addition for >> many >> > source connectors. >> > One clarification I had was regarding the topic settings that can be >> > configured. Is it limited to the setting exposed in the TopicSettings >> > interface? >> > >> > Thanks >> > Magesh >> > >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rha...@gmail.com> wrote: >> > >> > > Okay, after much delay let's try this again for AK 2.1. Has anyone >> found >> > > any concerns? Stephane suggested that we allow updating topic >> > > configurations (everything but partition count). I'm unconvinced that >> > it's >> > > worth the additional complexity in the implementation and the >> > documentation >> > > to explain the behavior. Changing several of the topic-specific >> > > configurations have significant impact on broker behavior / >> > functionality, >> > > so IMO we need to proceed more cautiously. >> > > >> > > Stephane, do you have a particular use case in mind for updating topic >> > > configurations on an existing topic? >> > > >> > > Randall >> > > >> > > >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> >> wrote: >> > > >> > > > The KIP deadline for 1.1 has already passed, but I'd like to restart >> > this >> > > > discussion so that we make the next release. I've not yet addressed >> the >> > > > previous comment about *existing* topics, but I'll try to do that >> over >> > > the >> > > > next few weeks. Any other comments/suggestions/questions? >> > > > >> > > > Best regards, >> > > > >> > > > Randall >> > > > >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> >> > wrote: >> > > > >> > > >> Oops. Yes, I meant “replication factor”. >> > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > > >> > >> > > >> > Randall: >> > > >> > bq. AdminClient currently allows changing the replication >> factory. >> > > >> > >> > > >> > By 'replication factory' did you mean 'replication factor' ? >> > > >> > >> > > >> > Cheers >> > > >> > >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com >> > >> > > >> wrote: >> > > >> >> >> > > >> >> Currently the KIP's scope is only topics that don't yet exist, >> and >> > we >> > > >> have >> > > >> >> to cognizant of race conditions between tasks with the same >> > > connector. >> > > >> I >> > > >> >> think it is worthwhile to consider whether the KIP's scope >> should >> > > >> expand to >> > > >> >> also address *existing* partitions, though it may not be >> > appropriate >> > > to >> > > >> >> have as much control when changing the topic settings for an >> > existing >> > > >> >> topic. For example, changing the number of partitions (which the >> > KIP >> > > >> >> considers a "topic-specific setting" even though technically it >> is >> > > not) >> > > >> >> shouldn't be done blindly due to the partitioning impacts, and >> IIRC >> > > you >> > > >> >> can't reduce them (which we could verify before applying). >> Also, I >> > > >> don't >> > > >> >> think the AdminClient currently allows changing the replication >> > > >> factory. I >> > > >> >> think changing the topic configs is less problematic both from >> what >> > > >> makes >> > > >> >> sense for connectors to verify/change and from what the >> AdminClient >> > > >> >> supports. >> > > >> >> >> > > >> >> Even if we decide that it's not appropriate to change the >> settings >> > on >> > > >> an >> > > >> >> existing topic, I do think it's advantageous to at least notify >> the >> > > >> >> connector (or task) prior to the first record sent to a given >> topic >> > > so >> > > >> that >> > > >> >> the connector can fail or issue a warning if it doesn't meet its >> > > >> >> requirements. >> > > >> >> >> > > >> >> Best regards, >> > > >> >> >> > > >> >> Randall >> > > >> >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < >> > > >> >> steph...@simplemachines.com.au> wrote: >> > > >> >> >> > > >> >>> Hi Randall, >> > > >> >>> >> > > >> >>> Thanks for the KIP. I like it >> > > >> >>> What happens when the target topic is already created but the >> > > configs >> > > >> do >> > > >> >>> not match? >> > > >> >>> i.e. wrong RF, num partitions, or missing / additional configs? >> > Will >> > > >> you >> > > >> >>> attempt to apply the necessary changes or throw an error? >> > > >> >>> >> > > >> >>> Thanks! >> > > >> >>> Stephane >> > > >> >>> >> > > >> >>> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < >> > > mathieu.fenn...@replicon.com >> > > >> > >> > > >> >>> wrote: >> > > >> >>> >> > > >> >>> Ah, yes, I see you a highlighted part that should've made >> this >> > > >> clear >> > > >> >>> to me the first read. :-) Much clearer now! >> > > >> >>> >> > > >> >>> By the way, enjoyed your Debezium talk in NYC. >> > > >> >>> >> > > >> >>> Looking forward to this Kafka Connect change; it will allow >> me >> > to >> > > >> >>> remove a post-deployment tool that I hacked together for the >> > > >> purpose >> > > >> >>> of ensuring auto-created topics have the right config. >> > > >> >>> >> > > >> >>> Mathieu >> > > >> >>> >> > > >> >>> >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch < >> > > rha...@gmail.com> >> > > >> >>> wrote: >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first >> > > >> >> configuration >> > > >> >>> rule >> > > >> >>>> whose regex matches will be applied, and no other rules will >> be >> > > >> >>> used. I've >> > > >> >>>> updated the KIP to try to make this more clear, but let me >> know >> > if >> > > >> >>> it's >> > > >> >>>> still not clear. >> > > >> >>>> >> > > >> >>>> Best regards, >> > > >> >>>> >> > > >> >>>> Randall >> > > >> >>>> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < >> > > >> >>>> mathieu.fenn...@replicon.com> wrote: >> > > >> >>>> >> > > >> >>>>> Hi Randall, >> > > >> >>>>> >> > > >> >>>>> Awesome, very much looking forward to this. >> > > >> >>>>> >> > > >> >>>>> It isn't 100% clear from the KIP how multiple config-based >> rules >> > > >> >>> would >> > > >> >>>>> be applied; it looks like the first configuration rule whose >> > regex >> > > >> >>>>> matches the topic name will be used, and no other rules will >> be >> > > >> >>>>> applied. Is that correct? (I wasn't sure if it might >> cascade >> > > >> >>>>> together multiple matching rules...) >> > > >> >>>>> >> > > >> >>>>> Looks great, >> > > >> >>>>> >> > > >> >>>>> Mathieu >> > > >> >>>>> >> > > >> >>>>> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch < >> > rha...@gmail.com> >> > > >> >>> wrote: >> > > >> >>>>>> Hi, all. >> > > >> >>>>>> >> > > >> >>>>>> We recently added the ability for Kafka Connect to create >> > > >> >>> *internal* >> > > >> >>>>> topics >> > > >> >>>>>> using the new AdminClient, but it still would be great if >> Kafka >> > > >> >>> Connect >> > > >> >>>>>> could do this for new topics that result from source >> connector >> > > >> >>> records. >> > > >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka >> Connect >> > > >> >>> should >> > > >> >>>>> allow >> > > >> >>>>>> source connectors to set topic-specific settings for new >> > > >> >> topics". >> > > >> >>>>>> >> > > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >> > > >> >>>>> set+topic-specific+settings+for+new+topics >> > > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >> > > >> >>>>> set+topic-specific+settings+for+new+topics>* >> > > >> >>>>>> >> > > >> >>>>>> Please take a look and provide feedback. Thanks! >> > > >> >>>>>> >> > > >> >>>>>> Best regards, >> > > >> >>>>>> >> > > >> >>>>>> Randall >> > > >> >>>>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >>> >> > > >> >> >> > > >> >> > > > >> > > > >> > > >> > >> >
