Hi Pushkar, I’ve been wondering if we should add writable tables to the Streams api. Can you explain more about your use case and how it would integrate with your application?
Incidentally, this would also help us provide more concrete advice. Thanks! John On Fri, May 1, 2020, at 15:28, Matthias J. Sax wrote: > Both stores sever a different purpose. > > Regular stores allow you to store state the application computes. > Writing into the changelog is a fault-tolerance mechanism. > > Global store hold "axially" data that is provided from "outside" of the > app. There is no changelog topic, but only the input topic (that is used > to re-create the global state). > > Local stores are sharded and updates are "sync" as they don't need to be > shared with anybody else. > > For global stores, as all instances need to be updated, updates are > async (we don't know when which instance will update it's own global > store replica). > > >> Say one stream thread updates the topic for global store and starts > >> processing next event wherein the processor tries to read the global store > >> which may not have been synced with the topic? > > Correct. There is no guarantee when the update to the global store will > be applied. As said, global stores are not designed to hold data the > application computes. > > > -Matthias > > > On 4/30/20 11:11 PM, Pushkar Deole wrote: > > thanks... will try with GlobalKTable. > > As a side question, I didn't really understand the significance of global > > state store which kind of works in a reverse way to local state store i.e. > > local state store is updated and then saved to changelog topic whereas in > > case of global state store the topic is updated first and then synced to > > global state store. Do these two work in sync i.e. the update to topic and > > global state store ? > > > > Say one stream thread updates the topic for global store and starts > > processing next event wherein the processor tries to read the global store > > which may not have been synced with the topic? > > > > On Fri, May 1, 2020 at 3:35 AM Matthias J. Sax <mj...@apache.org> wrote: > > > >> Yes. > >> > >> A `GlobalKTable` uses a global store internally. > >> > >> You can also use `StreamsBuilder.addGlobalStore()` or > >> `Topology.addGlobalStore()` to add a global store "manually". > >> > >> > >> -Matthias > >> > >> > >> On 4/30/20 7:42 AM, Pushkar Deole wrote: > >>> Thanks Matthias. > >>> Can you elaborate on the replicated caching layer part? > >>> When you say global stores, do you mean GlobalKTable created from a topic > >>> e.g. using StreamsBuilder.globalTable(String topic) method ? > >>> > >>> On Thu, Apr 30, 2020 at 12:44 PM Matthias J. Sax <mj...@apache.org> > >> wrote: > >>> > >>>> It's not possible to modify state store from "outside". > >>>> > >>>> If you want to build a "replicated caching layer", you could use global > >>>> stores and write into the corresponding topics to update all stores. Of > >>>> course, those updates would be async. > >>>> > >>>> > >>>> -Matthias > >>>> > >>>> On 4/29/20 10:52 PM, Pushkar Deole wrote: > >>>>> Hi All, > >>>>> > >>>>> I am wondering if this is possible: i have been asked to use state > >> stores > >>>>> as a general replicated cache among multiple instances of service > >>>> instances > >>>>> however the state store is created through streambuilder but is not > >>>>> actually modified through stream processor topology however it is to be > >>>>> modified from outside the stream topology. So, essentially, the state > >>>> store > >>>>> is just to be created from streambuilder and then to be used as an > >>>>> application level cache that will get replicated between application > >>>>> instances. Is this possible using state stores? > >>>>> > >>>>> Secondly, if possible, is this a good design approach? > >>>>> > >>>>> Appreciate your response since I don't know the internals of state > >>>> stores. > >>>>> > >>>> > >>>> > >>> > >> > >> > > > > > Attachments: > * signature.asc
