Hi, Just a friendly reminder to take a look at this KIP if you have the time.
I was thinking about broker vs. client default precedence, and I think it makes sense to keep the broker as the default used when both client-side and broker-side defaults are configured. The idea is that there would be pretty clear documentation, and that many systems with configurations that the client could not change would likely have the auto-create default off. (In cloud for example). It also seems like in most cases, the consumer config 'allow.auto.create.topics' was created to actually prevent the creation of topics, so the loss of creation functionality will not be a big problem. I'm happy to discuss any other compatibility problems or components of this KIP. Thank you, Justine On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <jols...@confluent.io> wrote: > Hello all, > > I was looking at this KIP again, and there is a decision I made that I > think is worth discussing. > > In the case where both the broker and producer's > 'auto.create.topics.enable' are set to true, we have to choose either the > broker configs or the producer configs for the replication > factor/partitions. > > Currently, the decision is to have the broker defaults take precedence. > (It is easier to do this in the implementation.) It also makes some sense > for this behavior to take precedence since this behavior already occurs as > the default. > > However, I was wondering if it would be odd for those who can only see the > producer side to set configs for replication factor/partitions and see > different results. Currently the documentation for the config states that > the config values are only used when the broker config is not enabled, but > this might not always be clear to the user. Changing the code to have the > producer's configurations take precedence is possible, but I was wondering > what everyone thought. > > Thank you, > Justine > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan <jols...@confluent.io> > wrote: > >> Just a quick update-- >> >> It seems that enabling both the broker and producer configs works fine, >> except that the broker configurations for partitions, replication factor >> take precedence. >> I don't know if that is something we would want to change, but I'll be >> updating the KIP for now to reflect this. Perhaps we would want to add more >> to the documentation of the the producer configs to clarify. >> >> Thank you, >> Justine >> >> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan <jols...@confluent.io> >> wrote: >> >>> Hi Colin, >>> >>> Thanks for looking at the KIP. I can definitely add to the title to make >>> it more clear. >>> >>> It makes sense that both configurations could be turned on since there >>> are many cases where the user can not control the server-side >>> configurations. I was a little concerned about how both interacting would >>> work out -- if there would be to many requests for new topics, for example. >>> But it since it does make sense to allow both configurations enabled, I >>> will test out some scenarios and I'll change the KIP to support this. >>> >>> I also agree with documentation about distinguishing the differences. I >>> was having some trouble with the wording but I like the phrases >>> "server-side" and "client-side." That's a good distinction I can use when >>> describing. >>> >>> I'll try to update the KIP soon keeping everyone's input in mind. >>> >>> Thanks, >>> Justine >>> >>> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <cmcc...@apache.org> wrote: >>> >>>> Hi Justine, >>>> >>>> Thanks for the KIP. This seems like a good step towards removing >>>> server-side topic auto-creation. >>>> >>>> We should add included "client-side" to the title of the KIP somewhere, >>>> to make it clear that we're talking about client-side auto creation. >>>> >>>> The KIP says: >>>> > In order to automatically create topics with the producer, the >>>> producer's >>>> > auto.create.topics.enable config must be set to true and the broker >>>> config should be set to false >>>> >>>> From a user's point of view, this seems counter-intuitive. In order to >>>> auto-create topics the broker's auto.create.topics.enable config should be >>>> set to false? It seems like the server-side auto-create is unrelated to >>>> the client-side auto-create. We could have both turned on (and I'm sure >>>> that in the real world, people will try this configuration...) There's no >>>> reason not to support this, I think. >>>> >>>> We should add some documentation explaining the difference between >>>> server-side and client-side auto-creation. Without documentation, an admin >>>> might think that they had disabled all forms of auto-creation by setting >>>> the -side setting to false-- but this is not the case, of course. >>>> >>>> best, >>>> Colin >>>> >>>> >>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: >>>> > Hi Dhruvil, >>>> > >>>> > Thanks for reading the KIP! >>>> > That was the general idea for deprecation. We would log a warning >>>> when the >>>> > config is enabled on the broker. >>>> > I also believe that there would be a change to documentation. >>>> > If there is anything else that should be done, please let me know! >>>> > >>>> > Justine >>>> > >>>> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah <dhru...@confluent.io> >>>> wrote: >>>> > >>>> > > Hi Justine, >>>> > > >>>> > > Thanks for the KIP, this is great! >>>> > > >>>> > > Could you add some more information about what deprecating the >>>> broker >>>> > > configuration means? Would we log a warning in the logs when auto >>>> topic >>>> > > creation is enabled on the broker, for example? >>>> > > >>>> > > Thanks, >>>> > > Dhruvil >>>> > > >>>> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < >>>> jols...@confluent.io> >>>> > > wrote: >>>> > > >>>> > > > Hello all, >>>> > > > >>>> > > > I'd like to start a discussion thread for KIP-487. >>>> > > > This KIP plans to deprecate the current system of auto-creating >>>> topics >>>> > > > through requests to the metadata and give the producer the >>>> ability to >>>> > > > automatically create topics instead. >>>> > > > >>>> > > > More information can be found here: >>>> > > > >>>> > > > >>>> > > >>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Automatic+Topic+Creation+on+Producer >>>> > > > >>>> > > > Thank you, >>>> > > > Justine Olshan >>>> > > > >>>> > > >>>> > >>>> >>>
