Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Hi all, I updated the KIP with the discussion from this thread. I left the warning message and deprecation out for now because we require the configuration be set to true (i.e. auto topic creation allowed) when using brokers older than 0.11.0. If there is no more feedback, I will start a VOTE

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

What is the status of this KIP? I think you can start a VOTE Dhruvil. -Matthias On 8/23/18 9:52 AM, Ismael Juma wrote: > Yeah, the reason why we want to deprecate the auto create functionality is > that it happens when a metadata request is done instead of when a write > operation happens. So,

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yeah, the reason why we want to deprecate the auto create functionality is that it happens when a metadata request is done instead of when a write operation happens. So, there's no reason to differentiate between the two. Ismael On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto wrote: > Ah, I just

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Ah, I just realized that as proposed this is only for the Java consumer client, correct? Would it be possible to make this a broker config, like the current one? Something like: auto.create.topics.enable=true # allow both producer and consumer to create auto.create.topics.enable=consumer #

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yup :) On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma wrote: > Andrew, one question: you are relying on auto topic creation for the > producer and that's why you can't just disable it? > > On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > > > Thanks for sharing Andrew! > > > > Ismael > > > >

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Andrew, one question: you are relying on auto topic creation for the producer and that's why you can't just disable it? On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > Thanks for sharing Andrew! > > Ismael > > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > >> We recently had a

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for sharing Andrew! Ismael On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > We recently had a pretty serious Kafka outage > < > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary > > > caused by a bug in one of our consumers that caused it to

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

We recently had a pretty serious Kafka outage caused by a bug in one of our consumers that caused it to create new topics in an infinite loop AKA a topic bomb! Having consumers restricted from creating

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Generally, I think positive configs (`allow` instead of `suppress`) are easier to understand. Ismael On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax wrote: > Thanks for the summary! > > We might want to add a diagram/table to the docs when we add this > feature (with whatever config name we

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for the summary! We might want to add a diagram/table to the docs when we add this feature (with whatever config name we choose) to explain how broker config `auto.create.topics.enable` and the consumer config work together. I think both options are equally easy to understand. "allow"

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

*"suppress" is the opposite of "allow", so setting suppress.auto.create.topics=false would mean that we do _not_ allow auto topic creation; when set to true, the server configuration will determine whether we allow automatic creation or not.* Sorry, I meant suppress.auto.create.topics=true above

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

To be clear, we will allow auto topic creation only when server config auto.create.topics.enable=true and consumer config allow.auto.create.topics=true; when either is false, we would not create the topic if it does not exist. "suppress" is the opposite of "allow", so setting

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

@Dhruvil: Yes, I had "suppress.auto.topic.creation" or "allow.auto.topic.creation" in mind. @Brandon: I think it would be "allow=false" or "suppress=true" to enable this feature. Ie, default values would be "allow=true" or "suppress=false". Not sure if this changes any preferences for you.

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

1 seems reasonable to me, would be good to hear other opinions. With regards to 2, yes, we discussed deprecating and eventually removing the server config. But we'd like to make it easier to get auto create behaviour on the producer (and maybe the consumer although that's less clear) before we do

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Hi Ismael, Thanks for the comments. Replies below. 1. We could throw an InvalidConfigurationException at run-time either when building the MetadataRequest or when an ApiVersions response is received. Because this is a configuration problem, I think users would likely see the exception on the

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

“allow=false” seems a bit more intuitive to me than “suppress=false” Brandon > On Aug 22, 2018, at 8:48 PM, Ted Yu wrote: > > We may also consider : > > "suppress.auto.topic.creation" > > or > > "allow.auto.topic.creation" > > w.r.t. suppress or allow, I don't have strong opinion either.

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

We may also consider : "suppress.auto.topic.creation" or "allow.auto.topic.creation" w.r.t. suppress or allow, I don't have strong opinion either. It's just a matter of choosing the proper default value. Cheers On Wed, Aug 22, 2018 at 6:00 PM Dhruvil Shah wrote: > Hi Matthias, > > Do you

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Hi Matthias, Do you mean something like "suppress.auto.create.topic"? I am leaning a bit towards "allow.auto.create.topics" but I don't have a strong preference either. Let's wait to hear if anyone else has an opinion on this. Thanks, Dhruvil On Tue, Aug 21, 2018 at 5:28 PM Matthias J. Sax

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for the KIP. A few questions/comments: 1. It seems hard to reason about if we just disregard the config for older brokers. Maybe we should throw an error if the brokers don't support it and let users explicitly change the config if they want to. 2. We probably want to switch the default

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for the KIP Dhruvil! I agree with Jason's comment. An alternative might be to use "suppress" what would revert the logic of "allow". Not sure which one is more intuitive and I am fine with both (no personal preference). Just wanted to mention it as an alternative. Don't have any further

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Hey Dhruvil, I would suggest using the verb "allow" rather than "enable. The consumer cannot enable auto topic creation because it is configured on the broker. All it can do is prevent it from happening if it is enabled. -Jason On Tue, Aug 21, 2018 at 3:56 PM, Dhruvil Shah wrote: > Hi, > > I

[DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Hi, I would like to start discussion on KIP-361 that proposes we add a consumer configuration to disable auto topic creation. Link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-361%3A+Add+Consumer+Configuration+to+Disable+Auto+Topic+Creation Suggestions and feedback are