Okay, after much delay let's try this again for AK 2.1. Has anyone found any concerns? Stephane suggested that we allow updating topic configurations (everything but partition count). I'm unconvinced that it's worth the additional complexity in the implementation and the documentation to explain the behavior. Changing several of the topic-specific configurations have significant impact on broker behavior / functionality, so IMO we need to proceed more cautiously.
Stephane, do you have a particular use case in mind for updating topic configurations on an existing topic? Randall On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> wrote: > The KIP deadline for 1.1 has already passed, but I'd like to restart this > discussion so that we make the next release. I've not yet addressed the > previous comment about *existing* topics, but I'll try to do that over the > next few weeks. Any other comments/suggestions/questions? > > Best regards, > > Randall > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> wrote: > >> Oops. Yes, I meant “replication factor”. >> >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> > >> > Randall: >> > bq. AdminClient currently allows changing the replication factory. >> > >> > By 'replication factory' did you mean 'replication factor' ? >> > >> > Cheers >> > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> >> wrote: >> >> >> >> Currently the KIP's scope is only topics that don't yet exist, and we >> have >> >> to cognizant of race conditions between tasks with the same connector. >> I >> >> think it is worthwhile to consider whether the KIP's scope should >> expand to >> >> also address *existing* partitions, though it may not be appropriate to >> >> have as much control when changing the topic settings for an existing >> >> topic. For example, changing the number of partitions (which the KIP >> >> considers a "topic-specific setting" even though technically it is not) >> >> shouldn't be done blindly due to the partitioning impacts, and IIRC you >> >> can't reduce them (which we could verify before applying). Also, I >> don't >> >> think the AdminClient currently allows changing the replication >> factory. I >> >> think changing the topic configs is less problematic both from what >> makes >> >> sense for connectors to verify/change and from what the AdminClient >> >> supports. >> >> >> >> Even if we decide that it's not appropriate to change the settings on >> an >> >> existing topic, I do think it's advantageous to at least notify the >> >> connector (or task) prior to the first record sent to a given topic so >> that >> >> the connector can fail or issue a warning if it doesn't meet its >> >> requirements. >> >> >> >> Best regards, >> >> >> >> Randall >> >> >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < >> >> steph...@simplemachines.com.au> wrote: >> >> >> >>> Hi Randall, >> >>> >> >>> Thanks for the KIP. I like it >> >>> What happens when the target topic is already created but the configs >> do >> >>> not match? >> >>> i.e. wrong RF, num partitions, or missing / additional configs? Will >> you >> >>> attempt to apply the necessary changes or throw an error? >> >>> >> >>> Thanks! >> >>> Stephane >> >>> >> >>> >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <mathieu.fenn...@replicon.com >> > >> >>> wrote: >> >>> >> >>> Ah, yes, I see you a highlighted part that should've made this >> clear >> >>> to me the first read. :-) Much clearer now! >> >>> >> >>> By the way, enjoyed your Debezium talk in NYC. >> >>> >> >>> Looking forward to this Kafka Connect change; it will allow me to >> >>> remove a post-deployment tool that I hacked together for the >> purpose >> >>> of ensuring auto-created topics have the right config. >> >>> >> >>> Mathieu >> >>> >> >>> >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> >> >>> wrote: >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first >> >> configuration >> >>> rule >> >>>> whose regex matches will be applied, and no other rules will be >> >>> used. I've >> >>>> updated the KIP to try to make this more clear, but let me know if >> >>> it's >> >>>> still not clear. >> >>>> >> >>>> Best regards, >> >>>> >> >>>> Randall >> >>>> >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < >> >>>> mathieu.fenn...@replicon.com> wrote: >> >>>> >> >>>>> Hi Randall, >> >>>>> >> >>>>> Awesome, very much looking forward to this. >> >>>>> >> >>>>> It isn't 100% clear from the KIP how multiple config-based rules >> >>> would >> >>>>> be applied; it looks like the first configuration rule whose regex >> >>>>> matches the topic name will be used, and no other rules will be >> >>>>> applied. Is that correct? (I wasn't sure if it might cascade >> >>>>> together multiple matching rules...) >> >>>>> >> >>>>> Looks great, >> >>>>> >> >>>>> Mathieu >> >>>>> >> >>>>> >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> >> >>> wrote: >> >>>>>> Hi, all. >> >>>>>> >> >>>>>> We recently added the ability for Kafka Connect to create >> >>> *internal* >> >>>>> topics >> >>>>>> using the new AdminClient, but it still would be great if Kafka >> >>> Connect >> >>>>>> could do this for new topics that result from source connector >> >>> records. >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka Connect >> >>> should >> >>>>> allow >> >>>>>> source connectors to set topic-specific settings for new >> >> topics". >> >>>>>> >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >> >>>>> set+topic-specific+settings+for+new+topics >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ >> >>>>> set+topic-specific+settings+for+new+topics>* >> >>>>>> >> >>>>>> Please take a look and provide feedback. Thanks! >> >>>>>> >> >>>>>> Best regards, >> >>>>>> >> >>>>>> Randall >> >>>>> >> >>> >> >>> >> >>> >> >>> >> >> >> > >
