Re: data structures used by GlobalKTable, KTable

By default, RocksDB is used. Ie, the following are the same: builder.globalTable("topic"); builder.globalTable( "topic", Materialized.as(Stores.persistentKeyValueStore("store-name")) ); -Matthias On 5/22/20 4:31 AM, Pushkar Deole wrote: > Matthias, > > I can see that when the Materialized

Re: data structures used by GlobalKTable, KTable

Matthias, I can see that when the Materialized parameter is provided with store-name, the store can be accessed through kafkaStreams.store(store-name). This returns ReadOnlyKeyValueStore which seems to hold the Map structure internally. So, the GlobalKTable API is just an abstraction and if I have

Re: data structures used by GlobalKTable, KTable

Your request is out-of-scope for the KIP. Sorry. Kafka Streams ships with an in-memory LRU-store though -- however, this store would never go back to the topic. It just caches bases on LRU and if data is evicted it's gone. Going back to the topic would not be feasible anyway, because there is no

Re: data structures used by GlobalKTable, KTable

If my thinking is correct then for some scenarios or use cases, the MRU for GlobalKTable might also work as a local store of data since the MRU will always store the data required by that application instance. On Sun, May 17, 2020 at 9:42 AM Pushkar Deole wrote: > Matthias, > > I would like to p

Re: data structures used by GlobalKTable, KTable

Matthias, I would like to provide a suggestion here. Please check if this can be converted into a KIP. Since GlobalKTable holds complete topic data, and when the store underneath is in-memory store then the data in memory can quickly grow to a large value. I think it would be good if while using G

Re: data structures used by GlobalKTable, KTable

thanks.. yes that would help On Thu, May 14, 2020 at 11:49 PM Matthias J. Sax wrote: > Yeah, the current API doesn't make it very clear how to do it. You can > set an in-memory like this: > > > builder.globalTable("topic", > Materialized.as(Stores.inMemoryKeyValueStore("store-name"))); > > > We

Re: data structures used by GlobalKTable, KTable

Yeah, the current API doesn't make it very clear how to do it. You can set an in-memory like this: > builder.globalTable("topic", > Materialized.as(Stores.inMemoryKeyValueStore("store-name"))); We are already working on an improved API via KIP-591: https://cwiki.apache.org/confluence/display/KA

Re: data structures used by GlobalKTable, KTable

Matthias, For GlobalKTable, I am looking at the APIs provided by StreamsBuilder and I don't see any option to mention in-memory store there: all these API documentation states that The resulting GlobalKTable

Re: data structures used by GlobalKTable, KTable

By default, RocksDB is used. You can also change it to use an in-memory store that is basically a HashMap. -Matthias On 5/12/20 10:16 AM, Pushkar Deole wrote: > Thanks Liam! > > On Tue, May 12, 2020, 15:12 Liam Clarke-Hutchinson < > liam.cla...@adscale.co.nz> wrote: > >> Hi Pushkar, >> >> Glob

Re: data structures used by GlobalKTable, KTable

Thanks Liam! On Tue, May 12, 2020, 15:12 Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Hi Pushkar, > > GlobalKTables and KTables can have whatever data structure you like, if you > provide the appropriate deserializers - for example, an Kafka Streams app I > maintain stores model d

Re: data structures used by GlobalKTable, KTable

Hi Pushkar, GlobalKTables and KTables can have whatever data structure you like, if you provide the appropriate deserializers - for example, an Kafka Streams app I maintain stores model data (exported to a topic per entity from Postgres via Kafka Connect's JDBC Source) as a GlobalKTable of Jackson

data structures used by GlobalKTable, KTable

Hello confluent team, Could you provide some information on what data structures are used internally by GlobalKTable and KTables. The application that I am working on has a requirement to read cached data from GlobalKTable on every incoming event, so the reads from GlobalKTable need to be efficien