Re: Explicitly creating topology topics in a streams app
Just FYI Streams explicitly disables auto topic creation. This is because we want to detect eg accidental deletion of internal topics, since that can/will result in data loss. Better to shut down and get someone's attention so they can try and revive the deleted topic or decide what to do. Not entirely sure which streams examples you were looking at but I'd guess the reason we set auto.create.topics.enable=true is not for the Streams app but for the consumers we use to push test data, so we don't have to explicitly create inuit topics On Thu, Nov 21, 2024 at 3:31 PM John D. Ament wrote: > Hi Paul > > > > On Thu, Nov 21, 2024 at 6:06 PM Brebner, Paul > wrote: > > > Hi John, > > > > I’m not a Kafka streams expert but have experimented a few times – I > > recall that Kafka Streams does need to create/use “internal topics” – and > > security has to be set on clients correctly from memory. > > This may help? > > > https://kafka.apache.org/23/documentation/streams/developer-guide/manage-topics > > And this > > > > > https://kafka.apache.org/23/documentation/streams/developer-guide/security.html#streams-developer-guide-security > > > Thanks, yeah, I've seen these two. The topics I'm referring to are the > internal topics around state store, and other similar internal use cases as > found at > > https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/dsl-topology-naming.html#L83-L95 > , which are generally (I believe) considered the non-user topics. > > To give a little more context, we run our production cluster with tight > security and prefer to disable the auto create topic setting, but for CI > purposes we're pretty open, hence didn't notice this error until > deployment. My hunch is that these internal topics require the auto create > setting set to true, but wanted to confirm before begging to flip it on > (and do the necessary security reviews and such). > > John > > > > > > > > Regards, Paul Brebner, NetApp > > > > From: John D. Ament > > Date: Friday, 22 November 2024 at 8:46 am > > To: users@kafka.apache.org > > Subject: Explicitly creating topology topics in a streams app > > [You don't often get email from johndam...@apache.org. Learn why this is > > important at https://aka.ms/LearnAboutSenderIdentification ] > > > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > > > > > > Hi, > > > > When I look at most stream examples I notice that they tend to include > the > > broker setting: > > > > auto.create.topics.enable=true > > > > My understanding is that this isn't usually recommended for production > > environments, so we have it off. We have started to play with Kafka > > Streams apps a bit but noticed that they fail in our production > > environments with errors around "UNKNOWN_TOPIC_OR_PARTITION". I suspect > > it's related to not having auto create enabled. > > > > Is there an option to force the client to create the topology topics that > > they use for Kafka Streams apps? Or is auto creation required? > > > > Thanks, > > > > John > > >
Re: Explicitly creating topology topics in a streams app
Hi Paul On Thu, Nov 21, 2024 at 6:06 PM Brebner, Paul wrote: > Hi John, > > I’m not a Kafka streams expert but have experimented a few times – I > recall that Kafka Streams does need to create/use “internal topics” – and > security has to be set on clients correctly from memory. > This may help? > https://kafka.apache.org/23/documentation/streams/developer-guide/manage-topics > And this > > https://kafka.apache.org/23/documentation/streams/developer-guide/security.html#streams-developer-guide-security Thanks, yeah, I've seen these two. The topics I'm referring to are the internal topics around state store, and other similar internal use cases as found at https://github.com/apache/kafka/blob/trunk/docs/streams/developer-guide/dsl-topology-naming.html#L83-L95 , which are generally (I believe) considered the non-user topics. To give a little more context, we run our production cluster with tight security and prefer to disable the auto create topic setting, but for CI purposes we're pretty open, hence didn't notice this error until deployment. My hunch is that these internal topics require the auto create setting set to true, but wanted to confirm before begging to flip it on (and do the necessary security reviews and such). John > > > Regards, Paul Brebner, NetApp > > From: John D. Ament > Date: Friday, 22 November 2024 at 8:46 am > To: users@kafka.apache.org > Subject: Explicitly creating topology topics in a streams app > [You don't often get email from johndam...@apache.org. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] > > EXTERNAL EMAIL - USE CAUTION when clicking links or attachments > > > > > Hi, > > When I look at most stream examples I notice that they tend to include the > broker setting: > > auto.create.topics.enable=true > > My understanding is that this isn't usually recommended for production > environments, so we have it off. We have started to play with Kafka > Streams apps a bit but noticed that they fail in our production > environments with errors around "UNKNOWN_TOPIC_OR_PARTITION". I suspect > it's related to not having auto create enabled. > > Is there an option to force the client to create the topology topics that > they use for Kafka Streams apps? Or is auto creation required? > > Thanks, > > John >
Re: Explicitly creating topology topics in a streams app
Hi John, I’m not a Kafka streams expert but have experimented a few times – I recall that Kafka Streams does need to create/use “internal topics” – and security has to be set on clients correctly from memory. This may help? https://kafka.apache.org/23/documentation/streams/developer-guide/manage-topics And this https://kafka.apache.org/23/documentation/streams/developer-guide/security.html#streams-developer-guide-security Regards, Paul Brebner, NetApp From: John D. Ament Date: Friday, 22 November 2024 at 8:46 am To: users@kafka.apache.org Subject: Explicitly creating topology topics in a streams app [You don't often get email from johndam...@apache.org. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] EXTERNAL EMAIL - USE CAUTION when clicking links or attachments Hi, When I look at most stream examples I notice that they tend to include the broker setting: auto.create.topics.enable=true My understanding is that this isn't usually recommended for production environments, so we have it off. We have started to play with Kafka Streams apps a bit but noticed that they fail in our production environments with errors around "UNKNOWN_TOPIC_OR_PARTITION". I suspect it's related to not having auto create enabled. Is there an option to force the client to create the topology topics that they use for Kafka Streams apps? Or is auto creation required? Thanks, John
