Michael, That makes sense to me! To clarify, in the current state of the KIP, the producer does not rely on the broker to autocreate--if the broker's config is disabled, then the producer can autocreate on its own with a create topic request (the same type of request the admin client uses). However, if both configs are enabled, the broker will autocreate through a metadata request before the producer gets a chance. Of course, the way to avoid this, is to do as you suggested, and set the "allow_auto_topic_creation" field to false.
I think the only thing we need to be careful with in this setup is without KIP 464, we can not use broker defaults for this topic. A user needs to specify the number of partition and replication factor in the config. An alternative to this is to have coded defaults for when these configs are unspecified, but it is not immediately apparent what these defaults should be. Thanks again for reading my KIP, Justine On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison <mickael.mai...@gmail.com> wrote: > Hi Justine, > > Thanks for the response! > In my opinion, it would be better if the producer did not rely at all > on the broker auto create feature as this is what we're aiming to > deprecate. When requesting metadata we can set the > "allow_auto_topic_creation" field to false to avoid the broker auto > creation. Then if the topic is not existing, send a > CreateTopicRequest. > > What do you think? > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan <jols...@confluent.io> > wrote: > > > > Currently the way it is implemented, the broker auto-creation > configuration > > takes precedence. The producer will not use the CreateTopics request. > > (Technically it can--but the topic will already be created through the > > broker, so it will never try to create the topic.) > > It is possible to change this however, and I'd be happy to discuss the > > benefits of this alternative. > > > > Thank you, > > Justine > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison < > mickael.mai...@gmail.com> > > wrote: > > > > > Hi Justine, > > > > > > Thanks for the KIP! > > > > > > In case auto creation is enabled on both the client and server, will > > > the producer still use the AdminClient (CreateTopics request) to > > > create topics? and not rely on the broker auto create. > > > I'm guessing the answer is yes but can you make it explicit. > > > > > > Thank you > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan <jols...@confluent.io> > > > wrote: > > > > > > > > Hi, > > > > Just a friendly reminder to take a look at this KIP if you have the > time. > > > > > > > > I was thinking about broker vs. client default precedence, and I > think it > > > > makes sense to keep the broker as the default used when both > client-side > > > > and broker-side defaults are configured. The idea is that there > would be > > > > pretty clear documentation, and that many systems with configurations > > > that > > > > the client could not change would likely have the auto-create default > > > off. > > > > (In cloud for example). > > > > > > > > It also seems like in most cases, the consumer config > > > > 'allow.auto.create.topics' was created to actually prevent the > creation > > > of > > > > topics, so the loss of creation functionality will not be a big > problem. > > > > > > > > I'm happy to discuss any other compatibility problems or components > of > > > > this KIP. > > > > > > > > Thank you, > > > > Justine > > > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan <jols...@confluent.io > > > > > wrote: > > > > > > > > > Hello all, > > > > > > > > > > I was looking at this KIP again, and there is a decision I made > that I > > > > > think is worth discussing. > > > > > > > > > > In the case where both the broker and producer's > > > > > 'auto.create.topics.enable' are set to true, we have to choose > either > > > the > > > > > broker configs or the producer configs for the replication > > > > > factor/partitions. > > > > > > > > > > Currently, the decision is to have the broker defaults take > precedence. > > > > > (It is easier to do this in the implementation.) It also makes some > > > sense > > > > > for this behavior to take precedence since this behavior already > > > occurs as > > > > > the default. > > > > > > > > > > However, I was wondering if it would be odd for those who can only > see > > > the > > > > > producer side to set configs for replication factor/partitions and > see > > > > > different results. Currently the documentation for the config > states > > > that > > > > > the config values are only used when the broker config is not > enabled, > > > but > > > > > this might not always be clear to the user. Changing the code to > have > > > the > > > > > producer's configurations take precedence is possible, but I was > > > wondering > > > > > what everyone thought. > > > > > > > > > > Thank you, > > > > > Justine > > > > > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan < > jols...@confluent.io> > > > > > wrote: > > > > > > > > > >> Just a quick update-- > > > > >> > > > > >> It seems that enabling both the broker and producer configs works > > > fine, > > > > >> except that the broker configurations for partitions, replication > > > factor > > > > >> take precedence. > > > > >> I don't know if that is something we would want to change, but > I'll be > > > > >> updating the KIP for now to reflect this. Perhaps we would want to > > > add more > > > > >> to the documentation of the the producer configs to clarify. > > > > >> > > > > >> Thank you, > > > > >> Justine > > > > >> > > > > >> On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan < > jols...@confluent.io> > > > > >> wrote: > > > > >> > > > > >>> Hi Colin, > > > > >>> > > > > >>> Thanks for looking at the KIP. I can definitely add to the title > to > > > make > > > > >>> it more clear. > > > > >>> > > > > >>> It makes sense that both configurations could be turned on since > > > there > > > > >>> are many cases where the user can not control the server-side > > > > >>> configurations. I was a little concerned about how both > interacting > > > would > > > > >>> work out -- if there would be to many requests for new topics, > for > > > example. > > > > >>> But it since it does make sense to allow both configurations > > > enabled, I > > > > >>> will test out some scenarios and I'll change the KIP to support > this. > > > > >>> > > > > >>> I also agree with documentation about distinguishing the > > > differences. I > > > > >>> was having some trouble with the wording but I like the phrases > > > > >>> "server-side" and "client-side." That's a good distinction I can > use > > > when > > > > >>> describing. > > > > >>> > > > > >>> I'll try to update the KIP soon keeping everyone's input in mind. > > > > >>> > > > > >>> Thanks, > > > > >>> Justine > > > > >>> > > > > >>> On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe <cmcc...@apache.org > > > > > wrote: > > > > >>> > > > > >>>> Hi Justine, > > > > >>>> > > > > >>>> Thanks for the KIP. This seems like a good step towards > removing > > > > >>>> server-side topic auto-creation. > > > > >>>> > > > > >>>> We should add included "client-side" to the title of the KIP > > > somewhere, > > > > >>>> to make it clear that we're talking about client-side auto > creation. > > > > >>>> > > > > >>>> The KIP says: > > > > >>>> > In order to automatically create topics with the producer, the > > > > >>>> producer's > > > > >>>> > auto.create.topics.enable config must be set to true and the > > > broker > > > > >>>> config should be set to false > > > > >>>> > > > > >>>> From a user's point of view, this seems counter-intuitive. In > > > order to > > > > >>>> auto-create topics the broker's auto.create.topics.enable config > > > should be > > > > >>>> set to false? It seems like the server-side auto-create is > > > unrelated to > > > > >>>> the client-side auto-create. We could have both turned on (and > I'm > > > sure > > > > >>>> that in the real world, people will try this configuration...) > > > There's no > > > > >>>> reason not to support this, I think. > > > > >>>> > > > > >>>> We should add some documentation explaining the difference > between > > > > >>>> server-side and client-side auto-creation. Without > documentation, > > > an admin > > > > >>>> might think that they had disabled all forms of auto-creation by > > > setting > > > > >>>> the -side setting to false-- but this is not the case, of > course. > > > > >>>> > > > > >>>> best, > > > > >>>> Colin > > > > >>>> > > > > >>>> > > > > >>>> On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > >>>> > Hi Dhruvil, > > > > >>>> > > > > > >>>> > Thanks for reading the KIP! > > > > >>>> > That was the general idea for deprecation. We would log a > warning > > > > >>>> when the > > > > >>>> > config is enabled on the broker. > > > > >>>> > I also believe that there would be a change to documentation. > > > > >>>> > If there is anything else that should be done, please let me > know! > > > > >>>> > > > > > >>>> > Justine > > > > >>>> > > > > > >>>> > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah < > > > dhru...@confluent.io> > > > > >>>> wrote: > > > > >>>> > > > > > >>>> > > Hi Justine, > > > > >>>> > > > > > > >>>> > > Thanks for the KIP, this is great! > > > > >>>> > > > > > > >>>> > > Could you add some more information about what deprecating > the > > > > >>>> broker > > > > >>>> > > configuration means? Would we log a warning in the logs when > > > auto > > > > >>>> topic > > > > >>>> > > creation is enabled on the broker, for example? > > > > >>>> > > > > > > >>>> > > Thanks, > > > > >>>> > > Dhruvil > > > > >>>> > > > > > > >>>> > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < > > > > >>>> jols...@confluent.io> > > > > >>>> > > wrote: > > > > >>>> > > > > > > >>>> > > > Hello all, > > > > >>>> > > > > > > > >>>> > > > I'd like to start a discussion thread for KIP-487. > > > > >>>> > > > This KIP plans to deprecate the current system of > > > auto-creating > > > > >>>> topics > > > > >>>> > > > through requests to the metadata and give the producer the > > > > >>>> ability to > > > > >>>> > > > automatically create topics instead. > > > > >>>> > > > > > > > >>>> > > > More information can be found here: > > > > >>>> > > > > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > >>>> > > > > > > > >>>> > > > Thank you, > > > > >>>> > > > Justine Olshan > > > > >>>> > > > > > > > >>>> > > > > > > >>>> > > > > > >>>> > > > > >>> > > > >
