Do you think we should support name-value pairs, too? On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <mage...@confluent.io> wrote:
> Randall, > > Thanks a lot for the KIP. I think this would be a great addition for many > source connectors. > One clarification I had was regarding the topic settings that can be > configured. Is it limited to the setting exposed in the TopicSettings > interface? > > Thanks > Magesh > > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rha...@gmail.com> wrote: > > > Okay, after much delay let's try this again for AK 2.1. Has anyone found > > any concerns? Stephane suggested that we allow updating topic > > configurations (everything but partition count). I'm unconvinced that > it's > > worth the additional complexity in the implementation and the > documentation > > to explain the behavior. Changing several of the topic-specific > > configurations have significant impact on broker behavior / > functionality, > > so IMO we need to proceed more cautiously. > > > > Stephane, do you have a particular use case in mind for updating topic > > configurations on an existing topic? > > > > Randall > > > > > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> wrote: > > > > > The KIP deadline for 1.1 has already passed, but I'd like to restart > this > > > discussion so that we make the next release. I've not yet addressed the > > > previous comment about *existing* topics, but I'll try to do that over > > the > > > next few weeks. Any other comments/suggestions/questions? > > > > > > Best regards, > > > > > > Randall > > > > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> > wrote: > > > > > >> Oops. Yes, I meant “replication factor”. > > >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > >> > > > >> > Randall: > > >> > bq. AdminClient currently allows changing the replication factory. > > >> > > > >> > By 'replication factory' did you mean 'replication factor' ? > > >> > > > >> > Cheers > > >> > > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> > > >> wrote: > > >> >> > > >> >> Currently the KIP's scope is only topics that don't yet exist, and > we > > >> have > > >> >> to cognizant of race conditions between tasks with the same > > connector. > > >> I > > >> >> think it is worthwhile to consider whether the KIP's scope should > > >> expand to > > >> >> also address *existing* partitions, though it may not be > appropriate > > to > > >> >> have as much control when changing the topic settings for an > existing > > >> >> topic. For example, changing the number of partitions (which the > KIP > > >> >> considers a "topic-specific setting" even though technically it is > > not) > > >> >> shouldn't be done blindly due to the partitioning impacts, and IIRC > > you > > >> >> can't reduce them (which we could verify before applying). Also, I > > >> don't > > >> >> think the AdminClient currently allows changing the replication > > >> factory. I > > >> >> think changing the topic configs is less problematic both from what > > >> makes > > >> >> sense for connectors to verify/change and from what the AdminClient > > >> >> supports. > > >> >> > > >> >> Even if we decide that it's not appropriate to change the settings > on > > >> an > > >> >> existing topic, I do think it's advantageous to at least notify the > > >> >> connector (or task) prior to the first record sent to a given topic > > so > > >> that > > >> >> the connector can fail or issue a warning if it doesn't meet its > > >> >> requirements. > > >> >> > > >> >> Best regards, > > >> >> > > >> >> Randall > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > > >> >> steph...@simplemachines.com.au> wrote: > > >> >> > > >> >>> Hi Randall, > > >> >>> > > >> >>> Thanks for the KIP. I like it > > >> >>> What happens when the target topic is already created but the > > configs > > >> do > > >> >>> not match? > > >> >>> i.e. wrong RF, num partitions, or missing / additional configs? > Will > > >> you > > >> >>> attempt to apply the necessary changes or throw an error? > > >> >>> > > >> >>> Thanks! > > >> >>> Stephane > > >> >>> > > >> >>> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > > mathieu.fenn...@replicon.com > > >> > > > >> >>> wrote: > > >> >>> > > >> >>> Ah, yes, I see you a highlighted part that should've made this > > >> clear > > >> >>> to me the first read. :-) Much clearer now! > > >> >>> > > >> >>> By the way, enjoyed your Debezium talk in NYC. > > >> >>> > > >> >>> Looking forward to this Kafka Connect change; it will allow me > to > > >> >>> remove a post-deployment tool that I hacked together for the > > >> purpose > > >> >>> of ensuring auto-created topics have the right config. > > >> >>> > > >> >>> Mathieu > > >> >>> > > >> >>> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch < > > rha...@gmail.com> > > >> >>> wrote: > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first > > >> >> configuration > > >> >>> rule > > >> >>>> whose regex matches will be applied, and no other rules will be > > >> >>> used. I've > > >> >>>> updated the KIP to try to make this more clear, but let me know > if > > >> >>> it's > > >> >>>> still not clear. > > >> >>>> > > >> >>>> Best regards, > > >> >>>> > > >> >>>> Randall > > >> >>>> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > > >> >>>> mathieu.fenn...@replicon.com> wrote: > > >> >>>> > > >> >>>>> Hi Randall, > > >> >>>>> > > >> >>>>> Awesome, very much looking forward to this. > > >> >>>>> > > >> >>>>> It isn't 100% clear from the KIP how multiple config-based rules > > >> >>> would > > >> >>>>> be applied; it looks like the first configuration rule whose > regex > > >> >>>>> matches the topic name will be used, and no other rules will be > > >> >>>>> applied. Is that correct? (I wasn't sure if it might cascade > > >> >>>>> together multiple matching rules...) > > >> >>>>> > > >> >>>>> Looks great, > > >> >>>>> > > >> >>>>> Mathieu > > >> >>>>> > > >> >>>>> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch < > rha...@gmail.com> > > >> >>> wrote: > > >> >>>>>> Hi, all. > > >> >>>>>> > > >> >>>>>> We recently added the ability for Kafka Connect to create > > >> >>> *internal* > > >> >>>>> topics > > >> >>>>>> using the new AdminClient, but it still would be great if Kafka > > >> >>> Connect > > >> >>>>>> could do this for new topics that result from source connector > > >> >>> records. > > >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect > > >> >>> should > > >> >>>>> allow > > >> >>>>>> source connectors to set topic-specific settings for new > > >> >> topics". > > >> >>>>>> > > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > >> >>>>> set+topic-specific+settings+for+new+topics > > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > >> >>>>> set+topic-specific+settings+for+new+topics>* > > >> >>>>>> > > >> >>>>>> Please take a look and provide feedback. Thanks! > > >> >>>>>> > > >> >>>>>> Best regards, > > >> >>>>>> > > >> >>>>>> Randall > > >> >>>>> > > >> >>> > > >> >>> > > >> >>> > > >> >>> > > >> >> > > >> > > > > > > > > >
