Re: KafkaStreams KTable#through not creating changelog topic

Thanks, based on this we will re-evaluate the use of internal topics. The main motivation for using the internal changelog topics was to avoid duplication of data and have an easy way to access the update stream of any state store. Best, Mikael On Fri, Nov 25, 2016 at 9:52 AM Michael Noll

Re: KafkaStreams KTable#through not creating changelog topic

Mikael, > Sure, I guess the topic is auto-created the first time I start the topology > and the second time its there already. It could be possible to create > topics up front for us, or even use an admin call from inside the code. Yes, that (i.e. you are running with auto-topic creation

Re: KafkaStreams KTable#through not creating changelog topic

Sure, I guess the topic is auto-created the first time I start the topology and the second time its there already. It could be possible to create topics up front for us, or even use an admin call from inside the code. That said, as a user, I think it would be great with a function in the Kafka

Re: KafkaStreams KTable#through not creating changelog topic

Mikeal, When you use `through(..)` topics are not created by KafkaStreams. You need to create them yourself before you run the application. Thanks, Damian On Thu, 24 Nov 2016 at 11:27 Mikael Högqvist wrote: > Yes, the naming is not an issue. > > I've tested this with the

Re: KafkaStreams KTable#through not creating changelog topic

Yes, the naming is not an issue. I've tested this with the topology described earlier. Every time I start the topology with a call to .through() that references a topic that does not exist, I get an exception from the UncaughtExceptionHandler: Uncaught exception

Re: KafkaStreams KTable#through not creating changelog topic

> 1) Create a state store AND the changelog > topic 2) follow the Kafka Streams naming convention for changelog topics. > Basically, I want to have a method that does what .through() is supposed to > do according to the documentation, but without the "topic" parameter. I understand what you are

Re: KafkaStreams KTable#through not creating changelog topic

Hi Michael, thanks for the extensive explanation, and yes it definitely helps with my understanding of through(). :) You guessed correctly that I'm doing some "shenanings" where I'm trying to derive the changelog of a state store from the state store name. This works perfectly fine with with a

Re: KafkaStreams KTable#through not creating changelog topic

> - Also, in case you want to do some shenanigans (like for some tooling you're building > around state stores/changelogs/interactive queries) such detecting all state store changelogs > by doing the equivalent of `ls *-changelog`, then this will miss changelogs of KTables that are > created by

Re: KafkaStreams KTable#through not creating changelog topic

Mikael, regarding your second question: > 2) Regarding the use case, the topology looks like this: > > .stream(...) > .aggregate(..., "store-1") > .mapValues(...) > .through(..., "store-2") The last operator above would, without "..." ellipsis, be sth like `KTable#through("through-topic",

Re: KafkaStreams KTable#through not creating changelog topic

Hi Eno, 1) Great :) 2) Yes, we are using the Interactive Queries to access the state stores. In addition, we access the changelogs to subscribe to updates. For this reason we need to know the changelog topic name. Thanks, Mikael On Tue, Nov 22, 2016 at 8:43 PM Eno Thereska

Re: KafkaStreams KTable#through not creating changelog topic

HI Mikael, 1) The JavaDoc looks incorrect, thanks for reporting. Matthias is looking into fixing it. I agree that it can be confusing to have topic names that are not what one would expect. 2) If your goal is to query/read from the state stores, you can use Interactive Queries to do that (you

Re: KafkaStreams KTable#through not creating changelog topic

Sorry for being unclear, i'll try again :) 1) The JavaDoc for through is not correct, it states that a changelog topic will be created for the state store. That is, if I would call it with through("topic", "a-store"), I would expect a kafka topic "my-app-id-a-store-changelog" to be created. 2)

Re: KafkaStreams KTable#through not creating changelog topic

I cannot completely follow what want to achieve. However, the JavaDoc for through() seems not to be correct to me. Using through() will not create an extra internal changelog topic with the described naming schema, because the topic specified in through() can be used for this (there is no point

Re: KafkaStreams KTable#through not creating changelog topic

Hi Mikael, If you perform an aggregate and thus create another KTable, that KTable will have a changelog topic (and a state store that you can query with Interactive Queris - but this is tangential). It is true that source KTables don't need to create another topic, since they already have

KafkaStreams KTable#through not creating changelog topic

Hi, in the documentation for KTable#through, it is stated that a new changelog topic will be created for the table. It also states that calling through is equivalent to calling #to followed by KStreamBuilder#table.