Randall: bq. AdminClient currently allows changing the replication factory.
By 'replication factory' did you mean 'replication factor' ? Cheers On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> wrote: > Currently the KIP's scope is only topics that don't yet exist, and we have > to cognizant of race conditions between tasks with the same connector. I > think it is worthwhile to consider whether the KIP's scope should expand to > also address *existing* partitions, though it may not be appropriate to > have as much control when changing the topic settings for an existing > topic. For example, changing the number of partitions (which the KIP > considers a "topic-specific setting" even though technically it is not) > shouldn't be done blindly due to the partitioning impacts, and IIRC you > can't reduce them (which we could verify before applying). Also, I don't > think the AdminClient currently allows changing the replication factory. I > think changing the topic configs is less problematic both from what makes > sense for connectors to verify/change and from what the AdminClient > supports. > > Even if we decide that it's not appropriate to change the settings on an > existing topic, I do think it's advantageous to at least notify the > connector (or task) prior to the first record sent to a given topic so that > the connector can fail or issue a warning if it doesn't meet its > requirements. > > Best regards, > > Randall > > On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek < > steph...@simplemachines.com.au> wrote: > > > Hi Randall, > > > > Thanks for the KIP. I like it > > What happens when the target topic is already created but the configs do > > not match? > > i.e. wrong RF, num partitions, or missing / additional configs? Will you > > attempt to apply the necessary changes or throw an error? > > > > Thanks! > > Stephane > > > > > > On 24/5/17, 5:59 am, "Mathieu Fenniak" <mathieu.fenn...@replicon.com> > > wrote: > > > > Ah, yes, I see you a highlighted part that should've made this clear > > to me the first read. :-) Much clearer now! > > > > By the way, enjoyed your Debezium talk in NYC. > > > > Looking forward to this Kafka Connect change; it will allow me to > > remove a post-deployment tool that I hacked together for the purpose > > of ensuring auto-created topics have the right config. > > > > Mathieu > > > > > > On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> > > wrote: > > > Thanks for the quick feedback, Mathieu. Yes, the first > configuration > > rule > > > whose regex matches will be applied, and no other rules will be > > used. I've > > > updated the KIP to try to make this more clear, but let me know if > > it's > > > still not clear. > > > > > > Best regards, > > > > > > Randall > > > > > > On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak < > > > mathieu.fenn...@replicon.com> wrote: > > > > > >> Hi Randall, > > >> > > >> Awesome, very much looking forward to this. > > >> > > >> It isn't 100% clear from the KIP how multiple config-based rules > > would > > >> be applied; it looks like the first configuration rule whose regex > > >> matches the topic name will be used, and no other rules will be > > >> applied. Is that correct? (I wasn't sure if it might cascade > > >> together multiple matching rules...) > > >> > > >> Looks great, > > >> > > >> Mathieu > > >> > > >> > > >> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> > > wrote: > > >> > Hi, all. > > >> > > > >> > We recently added the ability for Kafka Connect to create > > *internal* > > >> topics > > >> > using the new AdminClient, but it still would be great if Kafka > > Connect > > >> > could do this for new topics that result from source connector > > records. > > >> > I've outlined an approach to do this in "KIP-158 Kafka Connect > > should > > >> allow > > >> > source connectors to set topic-specific settings for new > topics". > > >> > > > >> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > >> set+topic-specific+settings+for+new+topics > > >> > <https://cwiki.apache.org/confluence/display/KAFKA/KIP- > > >> 158%3A+Kafka+Connect+should+allow+source+connectors+to+ > > >> set+topic-specific+settings+for+new+topics>* > > >> > > > >> > Please take a look and provide feedback. Thanks! > > >> > > > >> > Best regards, > > >> > > > >> > Randall > > >> > > > > > > > > >
