I was wondering if it would be much simpler to just do a pass-through so that we can support any topic setting added in Kafka without any code changes in connect. Since these are for topics that will have the actual data stream, users might possibly need more flexibility in terms of how the topics get created.
Thanks Magesh On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rha...@gmail.com> wrote: > Do you think we should support name-value pairs, too? > > On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <mage...@confluent.io> > wrote: > > > Randall, > > > > Thanks a lot for the KIP. I think this would be a great addition for many > > source connectors. > > One clarification I had was regarding the topic settings that can be > > configured. Is it limited to the setting exposed in the TopicSettings > > interface? > > > > Thanks > > Magesh > > > > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <rha...@gmail.com> wrote: > > > > > Okay, after much delay let's try this again for AK 2.1. Has anyone > found > > > any concerns? Stephane suggested that we allow updating topic > > > configurations (everything but partition count). I'm unconvinced that > > it's > > > worth the additional complexity in the implementation and the > > documentation > > > to explain the behavior. Changing several of the topic-specific > > > configurations have significant impact on broker behavior / > > functionality, > > > so IMO we need to proceed more cautiously. > > > > > > Stephane, do you have a particular use case in mind for updating topic > > > configurations on an existing topic? > > > > > > Randall > > > > > > > > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <rha...@gmail.com> > wrote: > > > > > > > The KIP deadline for 1.1 has already passed, but I'd like to restart > > this > > > > discussion so that we make the next release. I've not yet addressed > the > > > > previous comment about *existing* topics, but I'll try to do that > over > > > the > > > > next few weeks. Any other comments/suggestions/questions? > > > > > > > > Best regards, > > > > > > > > Randall > > > > > > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> > > wrote: > > > > > > > >> Oops. Yes, I meant “replication factor”. > > > >> > > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > >> > > > > >> > Randall: > > > >> > bq. AdminClient currently allows changing the replication factory. > > > >> > > > > >> > By 'replication factory' did you mean 'replication factor' ? > > > >> > > > > >> > Cheers > > > >> > > > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> > > > >> wrote: > > > >> >> > > > >> >> Currently the KIP's scope is only topics that don't yet exist, > and > > we > > > >> have > > > >> >> to cognizant of race conditions between tasks with the same > > > connector. > > > >> I > > > >> >> think it is worthwhile to consider whether the KIP's scope should > > > >> expand to > > > >> >> also address *existing* partitions, though it may not be > > appropriate > > > to > > > >> >> have as much control when changing the topic settings for an > > existing > > > >> >> topic. For example, changing the number of partitions (which the > > KIP > > > >> >> considers a "topic-specific setting" even though technically it > is > > > not) > > > >> >> shouldn't be done blindly due to the partitioning impacts, and > IIRC > > > you > > > >> >> can't reduce them (which we could verify before applying). Also, > I > > > >> don't > > > >> >> think the AdminClient currently allows changing the replication > > > >> factory. I > > > >> >> think changing the topic configs is less problematic both from > what > > > >> makes > > > >> >> sense for connectors to verify/change and from what the > AdminClient > > > >> >> supports. > > > >> >> > > > >> >> Even if we decide that it's not appropriate to change the > settings > > on > > > >> an > > > >> >> existing topic, I do think it's advantageous to at least notify > the > > > >> >> connector (or task) prior to the first record sent to a given > topic > > > so > > > >> that > > > >> >> the connector can fail or issue a warning if it doesn't meet its > > > >> >> requirements. > > > >> >> > > > >> >> Best regards, > > > >> >> > > > >> >> Randall > > > >> >> > > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > > > >> >> steph...@simplemachines.com.au> wrote: > > > >> >> > > > >> >>> Hi Randall, > > > >> >>> > > > >> >>> Thanks for the KIP. I like it > > > >> >>> What happens when the target topic is already created but the > > > configs > > > >> do > > > >> >>> not match? > > > >> >>> i.e. wrong RF, num partitions, or missing / additional configs? > > Will > > > >> you > > > >> >>> attempt to apply the necessary changes or throw an error? > > > >> >>> > > > >> >>> Thanks! > > > >> >>> Stephane > > > >> >>> > > > >> >>> > > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" < > > > mathieu.fenn...@replicon.com > > > >> > > > > >> >>> wrote: > > > >> >>> > > > >> >>> Ah, yes, I see you a highlighted part that should've made > this > > > >> clear > > > >> >>> to me the first read. :-) Much clearer now! > > > >> >>> > > > >> >>> By the way, enjoyed your Debezium talk in NYC. > > > >> >>> > > > >> >>> Looking forward to this Kafka Connect change; it will allow > me > > to > > > >> >>> remove a post-deployment tool that I hacked together for the > > > >> purpose > > > >> >>> of ensuring auto-created topics have the right config. > > > >> >>> > > > >> >>> Mathieu > > > >> >>> > > > >> >>> > > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch < > > > rha...@gmail.com> > > > >> >>> wrote: > > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the first > > > >> >> configuration > > > >> >>> rule > > > >> >>>> whose regex matches will be applied, and no other rules will be > > > >> >>> used. I've > > > >> >>>> updated the KIP to try to make this more clear, but let me know > > if > > > >> >>> it's > > > >> >>>> still not clear. > > > >> >>>> > > > >> >>>> Best regards, > > > >> >>>> > > > >> >>>> Randall > > > >> >>>> > > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > > > >> >>>> mathieu.fenn...@replicon.com> wrote: > > > >> >>>> > > > >> >>>>> Hi Randall, > > > >> >>>>> > > > >> >>>>> Awesome, very much looking forward to this. > > > >> >>>>> > > > >> >>>>> It isn't 100% clear from the KIP how multiple config-based > rules > > > >> >>> would > > > >> >>>>> be applied; it looks like the first configuration rule whose > > regex > > > >> >>>>> matches the topic name will be used, and no other rules will > be > > > >> >>>>> applied. Is that correct? (I wasn't sure if it might cascade > > > >> >>>>> together multiple matching rules...) > > > >> >>>>> > > > >> >>>>> Looks great, > > > >> >>>>> > > > >> >>>>> Mathieu > > > >> >>>>> > > > >> >>>>> > > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch < > > rha...@gmail.com> > > > >> >>> wrote: > > > >> >>>>>> Hi, all. > > > >> >>>>>> > > > >> >>>>>> We recently added the ability for Kafka Connect to create > > > >> >>> *internal* > > > >> >>>>> topics > > > >> >>>>>> using the new AdminClient, but it still would be great if > Kafka > > > >> >>> Connect > > > >> >>>>>> could do this for new topics that result from source > connector > > > >> >>> records. > > > >> >>>>>> I've outlined an approach to do this in "KIP-158 Kafka > Connect > > > >> >>> should > > > >> >>>>> allow > > > >> >>>>>> source connectors to set topic-specific settings for new > > > >> >> topics". > > > >> >>>>>> > > > >> >>>>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > > >> >>>>> set+topic-specific+settings+for+new+topics > > > >> >>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > > >> >>>>> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > > >> >>>>> set+topic-specific+settings+for+new+topics>* > > > >> >>>>>> > > > >> >>>>>> Please take a look and provide feedback. Thanks! > > > >> >>>>>> > > > >> >>>>>> Best regards, > > > >> >>>>>> > > > >> >>>>>> Randall > > > >> >>>>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >>> > > > >> >> > > > >> > > > > > > > > > > > > > >
