Depends on the use case, Global state comes with the technical debt of having to do your own GC, but comes with more control. You could implement the pattern above with a long FixedWindow as well, which will take care of the GC within the window bound.
Sorry, its not a yes / no answer :-) On Tue, 7 Apr 2020 at 09:03, Mohil Khare <[email protected]> wrote: > Thanks a lot Reza for your quick response. Yeah saving the data in an > external system after timer expiry makes sense. > So do you suggest using a global window for maintaining state ? > > Thanks and regards > Mohil > > On Mon, Apr 6, 2020 at 5:37 PM Reza Ardeshir Rokni <[email protected]> > wrote: > >> Are you able to make use of the following pattern? >> >> Store StateA-metadata until no activity for Duration X, you can use a >> Timer to check this, then expire the value, but store in an >> external system. If you get a record that does want this value after >> expiry, call out to the external system and store the value again in key >> StateA-metadata. >> >> Cheers >> >> On Tue, 7 Apr 2020 at 08:03, Mohil Khare <[email protected]> wrote: >> >>> Hello all, >>> We are attempting a implement a use case where beam (java sdk) reads two >>> kind of records from data stream like Kafka: >>> >>> 1. Records of type A containing key and corresponding metadata. >>> 2. Records of type B containing the same key, but no metadata. Beam then >>> needs to fill metadata for records of type B by doing a lookup for >>> metadata using keys received in records of type A. >>> >>> Idea is to save metadata or rather state for keys received in records of >>> type A and then do a lookup when records of type B are received. >>> I have implemented this using the "@State" construct of beam. However my >>> problem is that we don't know when keys should expire. I don't think >>> keeping a global window will be a good idea as there could be many keys >>> (may be millions over a period of time) to be saved in a state. >>> >>> What is the best way to achieve this? I was reading about RedisIO, but >>> found that it is still in the experimental stage. Can someone please >>> recommend any solution to achieve this. >>> >>> Thanks and regards >>> Mohil >>> >>> >>> >>> >>> >>>
