Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi Matthias, I have configured the GlobalKTable to stream from a topic and application is working fine, however during automated build test cases, sometimes I get an exception: I believe this could be because of race between actual topic creation and the service startup (since topic creation may

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Matthias, I realized that the exception and actual problem is totally different. The problem was the client was not set with SSL truststore while server is SSLenabled. I also found this open bug on kafka https://issues.apache.org/jira/browse/KAFKA-4493 After setting the SSL properties on stream,

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks... i will try increasing the memory in case you don't spot anything wrong with the code. Other service also have streams and global k table but they use spring-kafka, but i think that should not matter, and it should work with normal kafka-streams code unless i am missing some

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

There is no hook. Only a restore listener, but this one is only used during startup when the global store is loaded. It's not sure during regular processing. Depending on your usage, maybe you can switch to a global store instead of GlobalKTable? That way, you can implement a custom `Processor`

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Matthias, I tried with default store as well but getting same error, can you please check if I am initializing the global store in the right way: public void setupGlobalCacheTables(String theKafkaServers) { Properties props = new Properties();

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi Matthias, By the way, I used the in-memory global store and the service is giving out of memory error during startup. Unfortunately i don't have a stack trace now but when i got stack the first time, the error was coming somewhere from memorypool.allocate or similar kind of method. If i get

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Ok... got it... is there any hook that I can attach to the global k table or global store? What I mean here is I want to know when the global store is updated with data from topic in that case the hook that I specified should be invoked so i can do some activity like logging that, this will allow

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

For example it could be some "static" information, like a mapping from zip code to city name. Something that does usually not change over time. -Matthias On 5/25/20 9:55 PM, Pushkar Deole wrote: > Matthias, > > I am wondering what you mean by "Global store hold "axially" data that is >

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Matthias, I am wondering what you mean by "Global store hold "axially" data that is provided from "outside" of the app" will you be able to give some example use case here as to what you mean by axially data provided from outside app? On Sat, May 2, 2020 at 1:58 AM Matthias J. Sax wrote: >

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi Pushkar, I’m glad you’ve been able to work through the issues. The GlobalKTable does store the data in memory (or on disk, depending how you configure it). I think the in-memory version uses a TreeMap, which is logarithmic time access. I think you’ll find it sufficiently fast regardless.

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

John, I think I can get the cache structure modified to make use of GlobalKTable here so the data can be shared across. I could get information that the admin data will be uploaded well in advance before main events so the issue with 'missed joins' won't exists since by the time main events start

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hello John, Matthias Sorry for bothering you, however this is now getting crazier. Initially I was under the impression that the cache being hold by application is in the form of key/value where key is the instance of agentId (e.g. 10) and value will hold other attributes (and their respective

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi Pushkar, To answer your question about tuning the global store latency, I think the biggest impact thing you can do is to configure the consumer that loads the data for global stores. You can pass configs specifically to the global consumer with the prefix: “ global.consumer.” Regarding

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks John... appreciate your inputs and suggestions. I have been assigned recently to this task (of persisting the cache) and haven't been involved in original design and architecture and agree with all the issues you have highlighted. However, at this point, i don't think the application can be

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks John... what parameters would affect the latency in case GlobalKTable will be used and is there any configurations that could be tuned to minimize the latency of sync with input topic? On Mon, May 4, 2020 at 10:20 PM John Roesler wrote: > Hello Pushkar, > > Yes, that’s correct. The

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hello Pushkar, Yes, that’s correct. The operation you describe is currently not supported. If you want to keep the structure you described in place, I’d suggest using an external database for the admin objects. I’ll give another idea below. With your current architecture, I’m a little

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks John. Actually, this is a normal consumer-producer application wherein there are 2 consumers (admin consumer and main consumer) consuming messages from 2 different topics. One of the consumers consumes messages from a admin topic and populates data in a cache e.g. lets say agent with agent

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi Pushkar, I’ve been wondering if we should add writable tables to the Streams api. Can you explain more about your use case and how it would integrate with your application? Incidentally, this would also help us provide more concrete advice. Thanks! John On Fri, May 1, 2020, at 15:28,

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Both stores sever a different purpose. Regular stores allow you to store state the application computes. Writing into the changelog is a fault-tolerance mechanism. Global store hold "axially" data that is provided from "outside" of the app. There is no changelog topic, but only the input topic

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

thanks... will try with GlobalKTable. As a side question, I didn't really understand the significance of global state store which kind of works in a reverse way to local state store i.e. local state store is updated and then saved to changelog topic whereas in case of global state store the topic

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Yes. A `GlobalKTable` uses a global store internally. You can also use `StreamsBuilder.addGlobalStore()` or `Topology.addGlobalStore()` to add a global store "manually". -Matthias On 4/30/20 7:42 AM, Pushkar Deole wrote: > Thanks Matthias. > Can you elaborate on the replicated caching layer

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Thanks Matthias. Can you elaborate on the replicated caching layer part? When you say global stores, do you mean GlobalKTable created from a topic e.g. using StreamsBuilder.globalTable(String topic) method ? On Thu, Apr 30, 2020 at 12:44 PM Matthias J. Sax wrote: > It's not possible to modify

Re: can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

It's not possible to modify state store from "outside". If you want to build a "replicated caching layer", you could use global stores and write into the corresponding topics to update all stores. Of course, those updates would be async. -Matthias On 4/29/20 10:52 PM, Pushkar Deole wrote: > Hi

can kafka state stores be used as a application level cache by application to modify it from outside the stream topology?

Hi All, I am wondering if this is possible: i have been asked to use state stores as a general replicated cache among multiple instances of service instances however the state store is created through streambuilder but is not actually modified through stream processor topology however it is to be