Randall, Thanks a lot for the KIP. I think this would be a great addition for many source connectors. One clarification I had was regarding the topic settings that can be configured. Is it limited to the setting exposed in the TopicSettings interface?
Thanks Magesh On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rha...@gmail.com> wrote: > Okay, after much delay let's try this again for AK 2.1. Has anyone found > any concerns? Stephane suggested that we allow updating topic > configurations (everything but partition count). I'm unconvinced that it's > worth the additional complexity in the implementation and the documentation > to explain the behavior. Changing several of the topic-specific > configurations have significant impact on broker behavior / functionality, > so IMO we need to proceed more cautiously. > > Stephane, do you have a particular use case in mind for updating topic > configurations on an existing topic? > > Randall > > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> wrote: > > > The KIP deadline for 1.1 has already passed, but I'd like to restart this > > discussion so that we make the next release. I've not yet addressed the > > previous comment about *existing* topics, but I'll try to do that over > the > > next few weeks. Any other comments/suggestions/questions? > > > > Best regards, > > > > Randall > > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> wrote: > > > >> Oops. Yes, I meant “replication factor”. > >> > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> > > >> > Randall: > >> > bq. AdminClient currently allows changing the replication factory. > >> > > >> > By 'replication factory' did you mean 'replication factor' ? > >> > > >> > Cheers > >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> > >> wrote: > >> >> > >> >> Currently the KIP's scope is only topics that don't yet exist, and we > >> have > >> >> to cognizant of race conditions between tasks with the same > connector. > >> I > >> >> think it is worthwhile to consider whether the KIP's scope should > >> expand to > >> >> also address *existing* partitions, though it may not be appropriate > to > >> >> have as much control when changing the topic settings for an existing > >> >> topic. For example, changing the number of partitions (which the KIP > >> >> considers a "topic-specific setting" even though technically it is > not) > >> >> shouldn't be done blindly due to the partitioning impacts, and IIRC > you > >> >> can't reduce them (which we could verify before applying). Also, I > >> don't > >> >> think the AdminClient currently allows changing the replication > >> factory. I > >> >> think changing the topic configs is less problematic both from what > >> makes > >> >> sense for connectors to verify/change and from what the AdminClient > >> >> supports. > >> >> > >> >> Even if we decide that it's not appropriate to change the settings on > >> an > >> >> existing topic, I do think it's advantageous to at least notify the > >> >> connector (or task) prior to the first record sent to a given topic > so > >> that > >> >> the connector can fail or issue a warning if it doesn't meet its > >> >> requirements. > >> >> > >> >> Best regards, > >> >> > >> >> Randall > >> >> > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > >> >> steph...@simplemachines.com.au> wrote: > >> >> > >> >>> Hi Randall, > >> >>> > >> >>> Thanks for the KIP. I like it > >> >>> What happens when the target topic is already created but the > configs > >> do > >> >>> not match? > >> >>> i.e. wrong RF, num partitions, or missing / additional configs? Will > >> you > >> >>> attempt to apply the necessary changes or throw an error? > >> >>> > >> >>> Thanks! > >> >>> Stephane > >> >>> > >> >>> > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > mathieu.fenn...@replicon.com > >> > > >> >>> wrote: > >> >>> > >> >>> Ah, yes, I see you a highlighted part that should've made this > >> clear > >> >>> to me the first read. :-) Much clearer now! > >> >>> > >> >>> By the way, enjoyed your Debezium talk in NYC. > >> >>> > >> >>> Looking forward to this Kafka Connect change; it will allow me to > >> >>> remove a post-deployment tool that I hacked together for the > >> purpose > >> >>> of ensuring auto-created topics have the right config. > >> >>> > >> >>> Mathieu > >> >>> > >> >>> > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch < > rha...@gmail.com> > >> >>> wrote: > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first > >> >> configuration > >> >>> rule > >> >>>> whose regex matches will be applied, and no other rules will be > >> >>> used. I've > >> >>>> updated the KIP to try to make this more clear, but let me know if > >> >>> it's > >> >>>> still not clear. > >> >>>> > >> >>>> Best regards, > >> >>>> > >> >>>> Randall > >> >>>> > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > >> >>>> mathieu.fenn...@replicon.com> wrote: > >> >>>> > >> >>>>> Hi Randall, > >> >>>>> > >> >>>>> Awesome, very much looking forward to this. > >> >>>>> > >> >>>>> It isn't 100% clear from the KIP how multiple config-based rules > >> >>> would > >> >>>>> be applied; it looks like the first configuration rule whose regex > >> >>>>> matches the topic name will be used, and no other rules will be > >> >>>>> applied. Is that correct? (I wasn't sure if it might cascade > >> >>>>> together multiple matching rules...) > >> >>>>> > >> >>>>> Looks great, > >> >>>>> > >> >>>>> Mathieu > >> >>>>> > >> >>>>> > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> > >> >>> wrote: > >> >>>>>> Hi, all. > >> >>>>>> > >> >>>>>> We recently added the ability for Kafka Connect to create > >> >>> *internal* > >> >>>>> topics > >> >>>>>> using the new AdminClient, but it still would be great if Kafka > >> >>> Connect > >> >>>>>> could do this for new topics that result from source connector > >> >>> records. > >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect > >> >>> should > >> >>>>> allow > >> >>>>>> source connectors to set topic-specific settings for new > >> >> topics". > >> >>>>>> > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> >>>>> set+topic-specific+settings+for+new+topics > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >> >>>>> set+topic-specific+settings+for+new+topics>* > >> >>>>>> > >> >>>>>> Please take a look and provide feedback. Thanks! > >> >>>>>> > >> >>>>>> Best regards, > >> >>>>>> > >> >>>>>> Randall > >> >>>>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >> > >> > > > > >
