@randall it's more of a "safety net" and making sure an application runs against a topic that's properly configured or crashes. Not absolutely needed
On Mon., 27 Aug. 2018, 6:04 pm McCaig, Rhys, <rhys_mcc...@comcast.com> wrote: > Randall, > > This KIP looks great to me. As for _updating_ topic configs - It’s a nice > to have but certainly something that I could live without in order to get > this KIP implemented. (Its not something I would use in my current setup > but I can see some cases where it could be part of the workflow for > mirrored topics). > If it were to be included, I’d be happier to see it hidden behind a config > flag - (if topic already exists, can be an option to WARN/FAIL or change > the topic, where the default would be warn?) > > Cheers, > Rhys > > > On Aug 21, 2018, at 10:58 PM, Randall Hauch <rha...@gmail.com> wrote: > > > > Okay, after much delay let's try this again for AK 2.1. Has anyone found > > any concerns? Stephane suggested that we allow updating topic > > configurations (everything but partition count). I'm unconvinced that > it's > > worth the additional complexity in the implementation and the > documentation > > to explain the behavior. Changing several of the topic-specific > > configurations have significant impact on broker behavior / > functionality, > > so IMO we need to proceed more cautiously. > > > > Stephane, do you have a particular use case in mind for updating topic > > configurations on an existing topic? > > > > Randall > > > > > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> wrote: > > > >> The KIP deadline for 1.1 has already passed, but I'd like to restart > this > >> discussion so that we make the next release. I've not yet addressed the > >> previous comment about *existing* topics, but I'll try to do that over > the > >> next few weeks. Any other comments/suggestions/questions? > >> > >> Best regards, > >> > >> Randall > >> > >> On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> > wrote: > >> > >>> Oops. Yes, I meant “replication factor”. > >>> > >>>> On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >>>> > >>>> Randall: > >>>> bq. AdminClient currently allows changing the replication factory. > >>>> > >>>> By 'replication factory' did you mean 'replication factor' ? > >>>> > >>>> Cheers > >>>> > >>>>> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> > >>> wrote: > >>>>> > >>>>> Currently the KIP's scope is only topics that don't yet exist, and we > >>> have > >>>>> to cognizant of race conditions between tasks with the same > connector. > >>> I > >>>>> think it is worthwhile to consider whether the KIP's scope should > >>> expand to > >>>>> also address *existing* partitions, though it may not be appropriate > to > >>>>> have as much control when changing the topic settings for an existing > >>>>> topic. For example, changing the number of partitions (which the KIP > >>>>> considers a "topic-specific setting" even though technically it is > not) > >>>>> shouldn't be done blindly due to the partitioning impacts, and IIRC > you > >>>>> can't reduce them (which we could verify before applying). Also, I > >>> don't > >>>>> think the AdminClient currently allows changing the replication > >>> factory. I > >>>>> think changing the topic configs is less problematic both from what > >>> makes > >>>>> sense for connectors to verify/change and from what the AdminClient > >>>>> supports. > >>>>> > >>>>> Even if we decide that it's not appropriate to change the settings on > >>> an > >>>>> existing topic, I do think it's advantageous to at least notify the > >>>>> connector (or task) prior to the first record sent to a given topic > so > >>> that > >>>>> the connector can fail or issue a warning if it doesn't meet its > >>>>> requirements. > >>>>> > >>>>> Best regards, > >>>>> > >>>>> Randall > >>>>> > >>>>> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > >>>>> steph...@simplemachines.com.au> wrote: > >>>>> > >>>>>> Hi Randall, > >>>>>> > >>>>>> Thanks for the KIP. I like it > >>>>>> What happens when the target topic is already created but the > configs > >>> do > >>>>>> not match? > >>>>>> i.e. wrong RF, num partitions, or missing / additional configs? Will > >>> you > >>>>>> attempt to apply the necessary changes or throw an error? > >>>>>> > >>>>>> Thanks! > >>>>>> Stephane > >>>>>> > >>>>>> > >>>>>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > mathieu.fenn...@replicon.com > >>>> > >>>>>> wrote: > >>>>>> > >>>>>> Ah, yes, I see you a highlighted part that should've made this > >>> clear > >>>>>> to me the first read. :-) Much clearer now! > >>>>>> > >>>>>> By the way, enjoyed your Debezium talk in NYC. > >>>>>> > >>>>>> Looking forward to this Kafka Connect change; it will allow me to > >>>>>> remove a post-deployment tool that I hacked together for the > >>> purpose > >>>>>> of ensuring auto-created topics have the right config. > >>>>>> > >>>>>> Mathieu > >>>>>> > >>>>>> > >>>>>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com > > > >>>>>> wrote: > >>>>>>> Thanks for the quick feedback, Mathieu. Yes, the first > >>>>> configuration > >>>>>> rule > >>>>>>> whose regex matches will be applied, and no other rules will be > >>>>>> used. I've > >>>>>>> updated the KIP to try to make this more clear, but let me know if > >>>>>> it's > >>>>>>> still not clear. > >>>>>>> > >>>>>>> Best regards, > >>>>>>> > >>>>>>> Randall > >>>>>>> > >>>>>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > >>>>>>> mathieu.fenn...@replicon.com> wrote: > >>>>>>> > >>>>>>>> Hi Randall, > >>>>>>>> > >>>>>>>> Awesome, very much looking forward to this. > >>>>>>>> > >>>>>>>> It isn't 100% clear from the KIP how multiple config-based rules > >>>>>> would > >>>>>>>> be applied; it looks like the first configuration rule whose regex > >>>>>>>> matches the topic name will be used, and no other rules will be > >>>>>>>> applied. Is that correct? (I wasn't sure if it might cascade > >>>>>>>> together multiple matching rules...) > >>>>>>>> > >>>>>>>> Looks great, > >>>>>>>> > >>>>>>>> Mathieu > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> > >>>>>> wrote: > >>>>>>>>> Hi, all. > >>>>>>>>> > >>>>>>>>> We recently added the ability for Kafka Connect to create > >>>>>> *internal* > >>>>>>>> topics > >>>>>>>>> using the new AdminClient, but it still would be great if Kafka > >>>>>> Connect > >>>>>>>>> could do this for new topics that result from source connector > >>>>>> records. > >>>>>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect > >>>>>> should > >>>>>>>> allow > >>>>>>>>> source connectors to set topic-specific settings for new > >>>>> topics". > >>>>>>>>> > >>>>>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >>>>>>>> set+topic-specific+settings+for+new+topics > >>>>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >>>>>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > >>>>>>>> set+topic-specific+settings+for+new+topics>* > >>>>>>>>> > >>>>>>>>> Please take a look and provide feedback. Thanks! > >>>>>>>>> > >>>>>>>>> Best regards, > >>>>>>>>> > >>>>>>>>> Randall > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>> > >> > >> > >
