Hello all, Thank you for all the feedback!
It seems that one of the main issues is how the client-side auto-creation can act on its own and does not simply block or allow auto-creation as configured by the broker. I think I was a bit unclear about this, but the idea of this KIP is to eventually replace the functionality of the broker config. We don't want to check if the broker config is also enabled because the idea is that the broker config would not be enabled, and only specific producers/clients would be auto-creating topics. Some older clients require auto-topic creation for only the topics they need, and this KIP would make these clients compatible with brokers that disable autocreation. I now understand the worry about security and 'overriding' the broker's auto.create.topic.enable configuration. However, in the case I explained above, having the broker stop the producer would prevent the clients I described above from being able to create topics. (Basically not allowing the main point of creating this KIP.) I'm not sure to go about having such a security feature. Perhaps adding another config on the broker to prevent this that would be off by default but could be turned on? It would complicate things more, but I'm open to discussing. Thank you, Justine On Tue, Aug 6, 2019 at 11:47 PM Colin McCabe <cmcc...@apache.org> wrote: > On Tue, Aug 6, 2019, at 21:38, Harsha Ch wrote: > > Hi Colin, > > "Hmm... I'm not sure I follow. Users don't have to build their own > > tooling, right? They can use any of the shell scripts that we've shipped > > in the last few releases. For example, if any of your users run it, this > > shell script will delete all of the topics from your non-security-enabled > > cluster: > > > > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list > 2>/dev/null > > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 > --delete > > --topic > > > > They will need to fill in the correct bootstrap servers list, of course, > > not localhost. This deletion script will work on some pretty old > brokers, > > even back to the 0.10 releases. It seems a little odd to trust your > users > > with this power, but not trust them to avoid changing a particular > > configuration key." > > > > The above will blocked by the server if we set delete.topic.enable to > false > > and thats exactly what I am asking for. > > Hi Harsha, > > I was wondering if someone was going to bring up that configuration :) > > it's an interesting complication, but globally disabling topic deletion is > not very practical for most use-cases. > > In any case, there are plenty of other bad things that users with full > permissions can do that aren't blocked by any server configuration. For > example, they can delete every record in every topic. I can write a script > for that too, and there's no server configuration you can set to disable > it. Or I could simply create hundreds of thousands of topics, until > cluster performance becomes unacceptable (this will be even more of a > problem if someone configured delete.topic.enable as false). Or publish > bad data to every topic, etc. etc. > > The point I'm trying to make here is that you can't rely on these kind of > server-side configurations for security. At most, they're a way to set up > certain very simple policies. But the policies are so simple that they're > hardly ever useful any more. > > For example, if the problem you want to solve is that you want a user to > only be able to create 50 topics and not delete anyone else's topics, you > can solve that with a CreateTopicsPolicy that limits the number of topics, > and some ACLs. There's no combination of auto.create.topics.enable and > delete.topic.enable that will help here. > > > > > "The downside is that if we wanted to check a server side configuration > > before sending the create topics request, the code would be more complex. > > The behavior would also not be consistent with how topic auto-creation is > > handled in Kafka Streams." > > > > I am not sure why you need to check server side configuration before > > sending create topics request. A user enables producer side config to > > create topics. > > Producer sends a request to the broker and if the broker has > > auto.topic.create.enable to true (default) it will allow creation of > > topics. If it set to false it returns error back to the client. > > auto.topic.create.enable has never affected CreateTopicsRequest. If you > submit a CreateTopicsRequest and you are authorized, a topic will be > created, regardless of what the value of auto.topic.create.enable is. This > behavior goes back a long way, to about 2016 (Kafka 0.10.1, I think?) > > > I don't see how this behavior will be different in Kafka streams. By > > default server allows the topic creation and with this KIP, It will only > > allow creation of topic when both producer and server side are turned on. > > Its exactly the same behavior in KIP-361. > > I suggest running a streams job as a test. It creates the topics it needs > using AdminClient. The value of auto.topic.create.enable is not relevant. > Whether it is true or false, the topics will still be created. > > > > > "In general, it would be nice if we could keep the security and access > > control model simple and not introduce a lot of special cases and > > exceptions. Kafka has basically converged on using ACLs and > > CreateTopicPolicy classes to control who can create topics and where. > > Adding more knobs seems like a step backwards, especially when the > > proposed knobs don't work consistently across components, and don't > provide true > > security." This is not about access control at all. Shipping sane > defaults should > > be prioritized. > > I don't think the default is really the question here. I think everyone > agrees that the default for client-side auto-topic creation should be off. > > The scenarios that you're describing (such as dealing with a poorly > configured client that tries to create topics it should not) do seem to be > about access control. > > > We keep talking about CreateTopicPolicy and yet we don't have default > > one and asking every user of Kafka implement their own doesn't make > sense > > here. > > The point of CreateTopicPolicy is exactly that you can implement your own, > though. It's a way of customizing what the policy is in your specific > cluster. > > I agree that most people don't want to write a custom policy. But most of > those people are satisfied with just setting up ACLs to set up simple > policies like this user can create topics, that user can't, etc. That's > why I suggested adding support for ACLs to insecure clusters might be > helpful. > > Also, just as a side note, wouldn't it would be impossible for us to > specify a default CreateTopicsPolicy that did anything at this point anyway > without breaking backwards compatibility? Maybe I'm misunderstanding the > proposal. > > > I am asking about exact behavior that we shipped for consumer side > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation > > > > I still didn't get any response why this behavior shouldn't be exactly > like > > Kafka consumer and why do we want to have producer to overrider server > side > > config and while not allowing consumer to do so. > > We are not even allowing the same contract and this will cause more > > confusion from the users standpoint. > > Hmm. The consumer already has a way to override the server side > configuration. See KIP-361: Add Consumer Configuration to Disable Auto > Topic Creation. This lets the consumer skip auto-creating topics, even if > auto-creation is enabled on the broker. > > To be fair, the KIP should probably discuss why we don't want to implement > client-side topic creation in the consumer in "rejected alternatives." > Maybe Justine can add more context here in the KIP. > > The last time we talked about this, the reasoning was that we wanted to > eventually get rid of consumer-side auto-topic creation entirely, and so it > wasn't worth implementing an improved way of doing it. Producer side auto > topic-creation, in contrast, will probably stick around, although we'd like > to deprecate and remove the problematic server-side mechanism over time. > > best, > Colin > > > > > > > On Tue, Aug 6, 2019 at 9:09 PM Colin McCabe <cmcc...@apache.org> wrote: > > > > > On Tue, Aug 6, 2019, at 18:06, Harsha Ch wrote: > > > > Not sure how the AdminClient applies here, It is an external API and > > > > is not part of KafkaProducer so any user who updates to latest > version of > > > > Kafka with this feature get to create the topics. > > > > They have to build a tooling around AdminClient allow themselves to > > > create > > > > topics. > > > > > > Hi Harsha, > > > > > > Hmm... I'm not sure I follow. Users don't have to build their own > > > tooling, right? They can use any of the shell scripts that we've > shipped > > > in the last few releases. For example, if any of your users run it, > this > > > shell script will delete all of the topics from your > non-security-enabled > > > cluster: > > > > > > ./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list > 2>/dev/null > > > | xargs -l ./bin/kafka-topics.sh --bootstrap-server localhost:9092 > --delete > > > --topic > > > > > > They will need to fill in the correct bootstrap servers list, of > course, > > > not localhost. This deletion script will work on some pretty old > brokers, > > > even back to the 0.10 releases. It seems a little odd to trust your > users > > > with this power, but not trust them to avoid changing a particular > > > configuration key. > > > > > > > There is no behavior in Kafka producer that allowed users to > > > > delete the topics or delete the records. So citing them as an > > > > example doesn't makes sense in this context. > > > > > > I think Kafka Streams is relevant here. After all, it's software that > we > > > ship as part of the official Kafka release. And it auto-creates topics > > > even when auto.create.topics.enable is set to false on the broker. > > > > > > I think that auto.create.topics.enable was never intended as a security > > > setting to control access. It was intended as a way to disable the > > > broker-side auto-create feature specifically, because there were some > > > problems with that specific feature. Broker-side auto-creation is > > > frustrating because it's on by default, and because it can auto-create > > > topics even if the producer or consumer didn't explicitly ask for that. > > > Neither of those problems applies to this KIP: producers have to > > > specifically opt in, and it won't be on by default. Basically, we > think > > > that client-side auto-creation is actually a lot better-- hence this > KIP :) > > > > > > > But there is > > > > a functionality which allowed creation of topics if they don't exist > in > > > the > > > > cluster and this behavior could be controlled by a config on the > server > > > > side. Now with this KIP we are allowing producer to make that > decision > > > > without any gateway on the server via configs. Any user who is not > aware > > > of > > > > such changes > > > > can accidentally create these topics and we are essentially removing > a > > > > config that exists in brokers today to block this accidental > creation and > > > > allowing clients to take control. > > > > > > Again, I hope I'm not misinterpreting, but I don't see how this can be > > > turned on accidentally. The user would have to specifically turn this > on > > > in the producer by setting the configuration key. > > > > > > > I still didn't get any positives of not having server side > configs? > > > > if you want to turn them on and allow any client to create topics > set the > > > > default to true > > > > and allow users who doesn't want to have this feature let them turn > them > > > > off. It will be the exact behavior as it is today, as far as > producer is > > > > concerned. I am not > > > > understanding why not having server side configs to gateways such a > hard > > > > requirement and this behavior exists today. As far I am concerned > this > > > > breaks the backward compatibility. > > > > > > The downside is that if we wanted to check a server side configuration > > > before sending the create topics request, the code would be more > complex. > > > The behavior would also not be consistent with how topic auto-creation > is > > > handled in Kafka Streams. > > > > > > In general, it would be nice if we could keep the security and access > > > control model simple and not introduce a lot of special cases and > > > exceptions. Kafka has basically converged on using ACLs and > > > CreateTopicPolicy classes to control who can create topics and where. > > > Adding more knobs seems like a step backwards, especially when the > proposed > > > knobs don't work consistently across components, and don't provide true > > > security. > > > > > > Maybe we should support an insecure mode where there are still users > and > > > ACLs. That would help people who don't want to set up Kerberos, etc. > but > > > still want to protect against misconfigured clients or accidents. > Hadoop > > > has something like this, and I think it was useful. NFS also supported > > > (supports?) a mode where you just pass whatever user ID you want and > the > > > system believes you. These things clearly don't protect against > malicious > > > users, but they can help set up policies when needed. > > > > > > best, > > > Colin > > > > > > > > > > > Thanks, > > > > Harsha > > > > > > > > > > > > On Tue, Aug 6, 2019 at 4:02 PM Colin McCabe <cmcc...@apache.org> > wrote: > > > > > > > > > Hi Harsha, > > > > > > > > > > Thanks for taking a look. > > > > > > > > > > I'm not sure I follow the discussion about AdminClient. > > > KafkaAdminClient > > > > > has been around for about 2 years at this point as a public class. > > > There > > > > > are many programs that use it to automatically create topics. > Kafka > > > > > Streams does this, for example. If any of your users run Kafka > > > Streams, > > > > > they will be auto-creating topics, regardless of what setting you > use > > > for > > > > > auto.create.topics.enable. > > > > > > > > > > A big part of the annoyance of auto-topic creation right now is > that > > > it is > > > > > on by default. The new configuration proposed by KIP-487 wouldn't > be. > > > > > Users would have to explicitly opt in to the new behavior of > > > client-side > > > > > topic creation. If you run without security, you're already > putting a > > > huge > > > > > amount of trust in your users. For example, you trust them not to > > > delete > > > > > records with the kafka-delete-records.sh command, or delete topics > with > > > > > kafka-topics.sh. Trusting them not to set a certain config value > seems > > > > > minor in comparison, right? > > > > > > > > > > best, > > > > > Colin > > > > > > > > > > On Tue, Aug 6, 2019, at 10:49, Harsha Chintalapani wrote: > > > > > > Hi, > > > > > > Even with policies one needs to implement that, so for every > > > user who > > > > > > doesn't want a producer to create topics or have limits around > > > > > partitions & > > > > > > replication factor they have to implement a policy. > > > > > > The KIP is changing the behavior , it might not be > introducing > > > the > > > > > > new functionality but it will enable producers to override the > create > > > > > topic > > > > > > config settings on the broker. What I am asking for to provide a > > > config > > > > > > that will disable auto creation of topics and if its enabled set > some > > > > > sane > > > > > > defaults so that clients can create a topic with in those > limits. I > > > don't > > > > > > see how this not related to this KIP. > > > > > > If the server config options as I suggested doesn't interest > > > you at > > > > > > least have a default CreateTopicPolices in place. > > > > > > To give an example, In our environment we disable the > > > > > > auto.create.topic.enable and force users to go through a > centralized > > > > > > service as we want capture more details about the topic creation > and > > > > > > requirements. With this KIP, a producer can create a topic with > no > > > > > bounds. > > > > > > Another example max.message.size we define that at cluster level > > > and one > > > > > > can override max.messsage.size at topic level, any > misconfiguration > > > at > > > > > this > > > > > > will cause service degradation. Its not always about the rogue > > > clients, > > > > > > users can easily misconfigure and can cause an outage. > > > > > > Again we can talk about CreateTopicPolicy but without having a > > > default > > > > > > implementation and asking everyone to implement their own while > > > changing > > > > > > the behavior in producer doesn't make sense to me. > > > > > > > > > > > > Thanks, > > > > > > Harsha > > > > > > > > > > > > > > > > > > On Tue, Aug 06, 2019 at 7:41 AM, Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > > > > > > > > > > > Hi Harsha, > > > > > > > > > > > > > > I mentioned policies and the authorizer. For example, with > > > > > > > CreateTopicPolicy, you can implement the limits you describe. > If > > > you > > > > > have > > > > > > > ideas of how that should be improved, please submit a KIP. My > > > point is > > > > > that > > > > > > > this KIP is not introducing any new functionality with regards > to > > > what > > > > > > > rogue clients can do. It's using the existing protocol that is > > > already > > > > > > > exposed via the AdminClient. So, I don't think we need to > address > > > it in > > > > > > > this KIP. Does that make sense? > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > On Tue, Aug 6, 2019 at 7:13 AM Harsha Chintalapani < > > > ka...@harsha.io> > > > > > > > wrote: > > > > > > > > > > > > > > Ismael, > > > > > > > Sure AdminClient can do that and we should've shipped a config > or > > > use > > > > > the > > > > > > > existing one to block that. Not all users are yet to upgrade to > > > > > AdminClient > > > > > > > and start using that to cause issues yet. In shared > environment we > > > > > should > > > > > > > allow server to set sane defaults and not allow every client > to go > > > > > ahead > > > > > > > create random no.of topic/partitions and replication factor. > Even > > > if > > > > > the > > > > > > > users want to allow topic creation proposed in the KIP , it > makes > > > > > sense to > > > > > > > have some guards against the no.of partitions and replication > > > factor. > > > > > > > Authorizer is not always an answer to block requests and having > > > users > > > > > set > > > > > > > server side configs to protect a multi-tenant environment is > > > required. > > > > > In a > > > > > > > non-secure environment Authorizer is a blunt instrument either > you > > > end > > > > > up > > > > > > > blocking everyone or allowing everyone. > > > > > > > I am asking to have server side that allow clients to create > > > topics or > > > > > not > > > > > > > , if they are allowed set a ceiling on max no.of partitions and > > > > > > > replication-factor. > > > > > > > > > > > > > > -Harsha > > > > > > > > > > > > > > On Mon, Aug 5 2019 at 8:58 PM, <isma...@gmail.com> wrote: > > > > > > > > > > > > > > Harsha, > > > > > > > > > > > > > > Rogue clients can use the admin client to create topics and > > > partitions. > > > > > > > ACLs and policies can help in that case as well as this one. > > > > > > > > > > > > > > Ismael > > > > > > > > > > > > > > On Mon, Aug 5, 2019, 7:48 PM Harsha Chintalapani < > ka...@harsha.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > Thanks for the KIP. > > > > > > > "When server-side auto-creation is disabled, client-side > > > auto-creation > > > > > > > will try to use client-side configurations" > > > > > > > If I understand correctly, this KIP is removing any server-side > > > > > blocking > > > > > > > client auto creation of topic? > > > > > > > if so this will present potential issue of rogue client > creating > > > ton of > > > > > > > topic-partitions and potentially bringing down the service for > > > everyone > > > > > > > > > > > > > > or > > > > > > > > > > > > > > degrade the service itself. > > > > > > > By reading the KIP its not clear to me that there is a clear > way to > > > > > block > > > > > > > auto creation topics of all together from clients by server > side > > > > > config. > > > > > > > Server side configs of default topic, partitions should take > higher > > > > > > > precedence and client shouldn't be able to create a topic with > > > higher > > > > > > > > > > > > > > no.of > > > > > > > > > > > > > > partitions, replication than what server config specifies. > > > > > > > > > > > > > > Thanks, > > > > > > > Harsha > > > > > > > > > > > > > > On Mon, Aug 05, 2019 at 5:24 PM, Justine Olshan < > > > jols...@confluent.io> > > > > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > I made some changes to the KIP. Hopefully this configuration > change > > > > > > > > > > > > > > will > > > > > > > > > > > > > > make things a little clearer. > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > > > KIP-487%3A+Client-side+Automatic+Topic+Creation+on+Producer > > > > > > > > > > > > > > Please let me know if you have any feedback or questions! > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 1:44 PM Colin McCabe < > cmcc...@apache.org> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Mickael, > > > > > > > > > > > > > > I think you bring up a good point. It would be better if we > didn't > > > ever > > > > > > > have to set up client-side configuration for this feature, and > > > KIP-464 > > > > > > > would let us skip this entirely. > > > > > > > > > > > > > > On Wed, Jul 31, 2019, at 09:19, Justine Olshan wrote: > > > > > > > > > > > > > > Hi Mickael, > > > > > > > I agree that KIP-464 works on newer brokers, but I was a bit > > > worried > > > > > > > > > > > > > > how > > > > > > > > > > > > > > things would play out on older brokers that* do not *have KIP > 464 > > > > > > > > > > > > > > included. > > > > > > > > > > > > > > Is it enough to throw an error in this case when producer > configs > > > are > > > > > > > > > > > > > > not > > > > > > > > > > > > > > specified? > > > > > > > > > > > > > > I think the right thing to do would be to log an error message > in > > > the > > > > > > > client. We will need to have that capability in any case, to > cover > > > > > > > scenarios like the client trying to auto-create a topic that > they > > > don't > > > > > > > have permission to create. Or a client trying to create a > topic on > > > a > > > > > > > > > > > > > > broker > > > > > > > > > > > > > > so old that CreateTopicsRequest is not supported. > > > > > > > > > > > > > > The big downside to relying on KIP-464 is that it is a very > recent > > > > > > > > > > > > > > feature > > > > > > > > > > > > > > -- so recent that it hasn't even made its way to any official > > > Apache > > > > > > > release. It's scheduled for the upcoming 2.4 release in a few > > > months. > > > > > > > > > > > > > > So if you view this KIP as a step towards removing broker-side > > > > > > > auto-create, you might want to support older brokers just to > > > accelerate > > > > > > > adoption, and hasten the day when we can finally flip > broker-side > > > > > > > auto-create to off (or even remove it entirely). > > > > > > > > > > > > > > I have to agree, though, that having client-side > configurations for > > > > > > > > > > > > > > number > > > > > > > > > > > > > > of partitions and replication factor is messy. Maybe it would > be > > > worth > > > > > > > > > > > > > > it > > > > > > > > > > > > > > to restrict support to post-KIP-464 brokers, if we could avoid > > > creating > > > > > > > more configs. > > > > > > > > > > > > > > best, > > > > > > > Colin > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 9:10 AM Mickael Maison < > > > > > > > > > > > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > We can rely on KIP-464 which allows to omit the partition > count or > > > > > > > replication factor when creating a topic. In that case, the > broker > > > > > > > > > > > > > > defaults > > > > > > > > > > > > > > are used. > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 4:55 PM Justine Olshan < > > > jols...@confluent.io> > > > > > > > wrote: > > > > > > > > > > > > > > Michael, > > > > > > > That makes sense to me! > > > > > > > To clarify, in the current state of the KIP, the producer does > not > > > > > > > > > > > > > > rely > > > > > > > > > > > > > > on > > > > > > > > > > > > > > the broker to autocreate--if the broker's config is disabled, > then > > > > > > > > > > > > > > the > > > > > > > > > > > > > > producer can autocreate on its own with a create topic request > (the > > > > > > > > > > > > > > same > > > > > > > > > > > > > > type of request the admin client uses). > > > > > > > However, if both configs are enabled, the broker will > autocreate > > > > > > > > > > > > > > through > > > > > > > > > > > > > > a > > > > > > > > > > > > > > metadata request before the producer gets a chance. Of course, > the > > > way > > > > > > > > > > > > > > to > > > > > > > > > > > > > > avoid this, is to do as you suggested, and set > > > > > > > > > > > > > > the > > > > > > > > > > > > > > "allow_auto_topic_creation" field to false. > > > > > > > > > > > > > > I think the only thing we need to be careful with in this > setup is > > > > > > > > > > > > > > without > > > > > > > > > > > > > > KIP 464, we can not use broker defaults for this topic. A user > > > needs > > > > > > > > > > > > > > to > > > > > > > > > > > > > > specify the number of partition and replication factor in the > > > config. > > > > > > > > > > > > > > An > > > > > > > > > > > > > > alternative to this is to have coded defaults for when these > > > > > > > > > > > > > > configs > > > > > > > > > > > > > > are > > > > > > > > > > > > > > unspecified, but it is not immediately apparent what these > defaults > > > > > > > > > > > > > > should > > > > > > > > > > > > > > be. > > > > > > > > > > > > > > Thanks again for reading my KIP, > > > > > > > Justine > > > > > > > > > > > > > > On Wed, Jul 31, 2019 at 4:19 AM Mickael Maison < > > > > > > > > > > > > > > mickael.mai...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > Thanks for the response! > > > > > > > In my opinion, it would be better if the producer did not rely > at > > > > > > > > > > > > > > all > > > > > > > > > > > > > > on the broker auto create feature as this is what we're aiming > to > > > > > > > deprecate. When requesting metadata we can set the > > > > > > > "allow_auto_topic_creation" field to false to avoid the broker > auto > > > > > > > creation. Then if the topic is not existing, send a > > > CreateTopicRequest. > > > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > On Mon, Jul 29, 2019 at 6:34 PM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Currently the way it is implemented, the broker auto-creation > > > > > > > > > > > > > > configuration > > > > > > > > > > > > > > takes precedence. The producer will not use the CreateTopics > > > > > > > > > > > > > > request. > > > > > > > > > > > > > > (Technically it can--but the topic will already be created > > > > > > > > > > > > > > through > > > > > > > > > > > > > > the > > > > > > > > > > > > > > broker, so it will never try to create the topic.) It is > possible > > > to > > > > > > > change this however, and I'd be happy to > > > > > > > > > > > > > > discuss > > > > > > > > > > > > > > the > > > > > > > > > > > > > > benefits of this alternative. > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Mon, Jul 29, 2019 at 10:26 AM Mickael Maison < > > > > > > > > > > > > > > mickael.mai...@gmail.com> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > Thanks for the KIP! > > > > > > > > > > > > > > In case auto creation is enabled on both the client and server, > > > > > > > > > > > > > > will > > > > > > > > > > > > > > the producer still use the AdminClient (CreateTopics request) > > > > > > > > > > > > > > to > > > > > > > > > > > > > > create topics? and not rely on the broker auto create. I'm > > > guessing the > > > > > > > answer is yes but can you make it explicit. > > > > > > > > > > > > > > Thank you > > > > > > > > > > > > > > On Wed, Jul 24, 2019 at 6:23 PM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi, > > > > > > > Just a friendly reminder to take a look at this KIP if you > > > > > > > > > > > > > > have > > > > > > > > > > > > > > the > > > > > > > > > > > > > > time. > > > > > > > > > > > > > > I was thinking about broker vs. client default precedence, > > > > > > > > > > > > > > and I > > > > > > > > > > > > > > think it > > > > > > > > > > > > > > makes sense to keep the broker as the default used when both > > > > > > > > > > > > > > client-side > > > > > > > > > > > > > > and broker-side defaults are configured. The idea is that > > > > > > > > > > > > > > there > > > > > > > > > > > > > > would be > > > > > > > > > > > > > > pretty clear documentation, and that many systems with > > > > > > > > > > > > > > configurations > > > > > > > > > > > > > > that > > > > > > > > > > > > > > the client could not change would likely have the auto-create > > > > > > > > > > > > > > default > > > > > > > > > > > > > > off. > > > > > > > > > > > > > > (In cloud for example). > > > > > > > > > > > > > > It also seems like in most cases, the consumer config > > > > > > > 'allow.auto.create.topics' was created to actually prevent > > > > > > > > > > > > > > the > > > > > > > > > > > > > > creation > > > > > > > > > > > > > > of > > > > > > > > > > > > > > topics, so the loss of creation functionality will not be a > > > > > > > > > > > > > > big > > > > > > > > > > > > > > problem. > > > > > > > > > > > > > > I'm happy to discuss any other compatibility problems or > > > > > > > > > > > > > > components > > > > > > > > > > > > > > of > > > > > > > > > > > > > > this KIP. > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Wed, Jul 17, 2019 at 9:11 AM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > I was looking at this KIP again, and there is a decision I > > > > > > > > > > > > > > made > > > > > > > > > > > > > > that I > > > > > > > > > > > > > > think is worth discussing. > > > > > > > > > > > > > > In the case where both the broker and producer's > > > > > > > 'auto.create.topics.enable' are set to true, we have to > > > > > > > > > > > > > > choose > > > > > > > > > > > > > > either > > > > > > > > > > > > > > the > > > > > > > > > > > > > > broker configs or the producer configs for the replication > > > > > > > factor/partitions. > > > > > > > > > > > > > > Currently, the decision is to have the broker defaults take > > > > > > > > > > > > > > precedence. > > > > > > > > > > > > > > (It is easier to do this in the implementation.) It also > > > > > > > > > > > > > > makes > > > > > > > > > > > > > > some > > > > > > > > > > > > > > sense > > > > > > > > > > > > > > for this behavior to take precedence since this behavior > > > > > > > > > > > > > > already > > > > > > > > > > > > > > occurs as > > > > > > > > > > > > > > the default. > > > > > > > > > > > > > > However, I was wondering if it would be odd for those who > > > > > > > > > > > > > > can > > > > > > > > > > > > > > only > > > > > > > > > > > > > > see > > > > > > > > > > > > > > the > > > > > > > > > > > > > > producer side to set configs for replication > > > > > > > > > > > > > > factor/partitions > > > > > > > > > > > > > > and > > > > > > > > > > > > > > see > > > > > > > > > > > > > > different results. Currently the documentation for the > > > > > > > > > > > > > > config > > > > > > > > > > > > > > states > > > > > > > > > > > > > > that > > > > > > > > > > > > > > the config values are only used when the broker config is > > > > > > > > > > > > > > not > > > > > > > > > > > > > > enabled, > > > > > > > > > > > > > > but > > > > > > > > > > > > > > this might not always be clear to the user. Changing the > > > > > > > > > > > > > > code > > > > > > > > > > > > > > to > > > > > > > > > > > > > > have > > > > > > > > > > > > > > the > > > > > > > > > > > > > > producer's configurations take precedence is possible, but > > > > > > > > > > > > > > I > > > > > > > > > > > > > > was > > > > > > > > > > > > > > wondering > > > > > > > > > > > > > > what everyone thought. > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Fri, Jul 12, 2019 at 2:49 PM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Just a quick update-- > > > > > > > > > > > > > > It seems that enabling both the broker and producer > > > > > > > > > > > > > > configs > > > > > > > > > > > > > > works > > > > > > > > > > > > > > fine, > > > > > > > > > > > > > > except that the broker configurations for partitions, > > > > > > > > > > > > > > replication > > > > > > > > > > > > > > factor > > > > > > > > > > > > > > take precedence. > > > > > > > I don't know if that is something we would want to > > > > > > > > > > > > > > change, but > > > > > > > > > > > > > > I'll be > > > > > > > > > > > > > > updating the KIP for now to reflect this. Perhaps we would > > > > > > > > > > > > > > want to > > > > > > > > > > > > > > add more > > > > > > > > > > > > > > to the documentation of the the producer configs to > > > > > > > > > > > > > > clarify. > > > > > > > > > > > > > > Thank you, > > > > > > > Justine > > > > > > > > > > > > > > On Fri, Jul 12, 2019 at 9:28 AM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Colin, > > > > > > > > > > > > > > Thanks for looking at the KIP. I can definitely add to > > > > > > > > > > > > > > the > > > > > > > > > > > > > > title > > > > > > > > > > > > > > to > > > > > > > > > > > > > > make > > > > > > > > > > > > > > it more clear. > > > > > > > > > > > > > > It makes sense that both configurations could be turned > > > > > > > > > > > > > > on > > > > > > > > > > > > > > since > > > > > > > > > > > > > > there > > > > > > > > > > > > > > are many cases where the user can not control the > > > > > > > > > > > > > > server-side > > > > > > > > > > > > > > configurations. I was a little concerned about how both > > > > > > > > > > > > > > interacting > > > > > > > > > > > > > > would > > > > > > > > > > > > > > work out -- if there would be to many requests for new > > > > > > > > > > > > > > topics, > > > > > > > > > > > > > > for > > > > > > > > > > > > > > example. > > > > > > > > > > > > > > But it since it does make sense to allow both > > > > > > > > > > > > > > configurations > > > > > > > > > > > > > > enabled, I > > > > > > > > > > > > > > will test out some scenarios and I'll change the KIP to > > > > > > > > > > > > > > support > > > > > > > > > > > > > > this. > > > > > > > > > > > > > > I also agree with documentation about distinguishing the > > > > > > > > > > > > > > differences. I > > > > > > > > > > > > > > was having some trouble with the wording but I like the > > > > > > > > > > > > > > phrases > > > > > > > > > > > > > > "server-side" and "client-side." That's a good > > > > > > > > > > > > > > distinction I > > > > > > > > > > > > > > can > > > > > > > > > > > > > > use > > > > > > > > > > > > > > when > > > > > > > > > > > > > > describing. > > > > > > > > > > > > > > I'll try to update the KIP soon keeping everyone's input > > > > > > > > > > > > > > in > > > > > > > > > > > > > > mind. > > > > > > > > > > > > > > Thanks, > > > > > > > Justine > > > > > > > > > > > > > > On Thu, Jul 11, 2019 at 5:39 PM Colin McCabe < > > > > > > > > > > > > > > cmcc...@apache.org > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > Thanks for the KIP. This seems like a good step towards > > > > > > > > > > > > > > removing > > > > > > > > > > > > > > server-side topic auto-creation. > > > > > > > > > > > > > > We should add included "client-side" to the title of > > > > > > > > > > > > > > the KIP > > > > > > > > > > > > > > somewhere, > > > > > > > > > > > > > > to make it clear that we're talking about client-side > > > > > > > > > > > > > > auto > > > > > > > > > > > > > > creation. > > > > > > > > > > > > > > The KIP says: > > > > > > > > > > > > > > In order to automatically create topics with the > > > > > > > > > > > > > > producer, the > > > > > > > > > > > > > > producer's > > > > > > > > > > > > > > auto.create.topics.enable config must be set to true > > > > > > > > > > > > > > and > > > > > > > > > > > > > > the > > > > > > > > > > > > > > broker > > > > > > > > > > > > > > config should be set to false > > > > > > > > > > > > > > From a user's point of view, this seems > > > > > > > > > > > > > > counter-intuitive. > > > > > > > > > > > > > > In > > > > > > > > > > > > > > order to > > > > > > > > > > > > > > auto-create topics the broker's > > > > > > > > > > > > > > auto.create.topics.enable > > > > > > > > > > > > > > config > > > > > > > > > > > > > > should be > > > > > > > > > > > > > > set to false? It seems like the server-side > > > > > > > > > > > > > > auto-create is > > > > > > > > > > > > > > unrelated to > > > > > > > > > > > > > > the client-side auto-create. We could have both turned > > > > > > > > > > > > > > on > > > > > > > > > > > > > > (and > > > > > > > > > > > > > > I'm > > > > > > > > > > > > > > sure > > > > > > > > > > > > > > that in the real world, people will try this > > > > > > > > > > > > > > configuration...) > > > > > > > > > > > > > > There's no > > > > > > > > > > > > > > reason not to support this, I think. > > > > > > > > > > > > > > We should add some documentation explaining the > > > > > > > > > > > > > > difference > > > > > > > > > > > > > > between > > > > > > > > > > > > > > server-side and client-side auto-creation. Without > > > > > > > > > > > > > > documentation, > > > > > > > > > > > > > > an admin > > > > > > > > > > > > > > might think that they had disabled all forms of > > > > > > > > > > > > > > auto-creation by > > > > > > > > > > > > > > setting > > > > > > > > > > > > > > the -side setting to false-- but this is not the case, > > > > > > > > > > > > > > of > > > > > > > > > > > > > > course. > > > > > > > > > > > > > > best, > > > > > > > Colin > > > > > > > > > > > > > > On Thu, Jul 11, 2019, at 16:22, Justine Olshan wrote: > > > > > > > > > > > > > > Hi Dhruvil, > > > > > > > > > > > > > > Thanks for reading the KIP! > > > > > > > That was the general idea for deprecation. We would > > > > > > > > > > > > > > log a > > > > > > > > > > > > > > warning > > > > > > > > > > > > > > when the > > > > > > > > > > > > > > config is enabled on the broker. > > > > > > > I also believe that there would be a change to > > > > > > > > > > > > > > documentation. > > > > > > > > > > > > > > If there is anything else that should be done, please > > > > > > > > > > > > > > let > > > > > > > > > > > > > > me > > > > > > > > > > > > > > know! > > > > > > > > > > > > > > Justine > > > > > > > > > > > > > > On Thu, Jul 11, 2019 at 4:17 PM Dhruvil Shah < > > > > > > > > > > > > > > dhru...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hi Justine, > > > > > > > > > > > > > > Thanks for the KIP, this is great! > > > > > > > > > > > > > > Could you add some more information about what > > > > > > > > > > > > > > deprecating > > > > > > > > > > > > > > the > > > > > > > > > > > > > > broker > > > > > > > > > > > > > > configuration means? Would we log a warning in the > > > > > > > > > > > > > > logs > > > > > > > > > > > > > > when > > > > > > > > > > > > > > auto > > > > > > > > > > > > > > topic > > > > > > > > > > > > > > creation is enabled on the broker, for example? > > > > > > > > > > > > > > Thanks, > > > > > > > Dhruvil > > > > > > > > > > > > > > On Thu, Jul 11, 2019 at 10:28 AM Justine Olshan < > > > > > > > > > > > > > > jols...@confluent.io> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > Hello all, > > > > > > > > > > > > > > I'd like to start a discussion thread for KIP-487. This KIP > plans > > > to > > > > > > > deprecate the current system of > > > > > > > > > > > > > > auto-creating > > > > > > > > > > > > > > topics > > > > > > > > > > > > > > through requests to the metadata and give the > > > > > > > > > > > > > > producer the > > > > > > > > > > > > > > ability to > > > > > > > > > > > > > > automatically create topics instead. > > > > > > > > > > > > > > More information can be found here: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > > > KIP-487%3A+Automatic+Topic+Creation+on+Producer > > > > > > > > > > > > > > Thank you, > > > > > > > Justine Olshan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
