No ... that's a valid answer. Since I wanted to have a long window size per key and since we can't use state with session windows, I am using a sliding window for let's say 72 hrs which starts every hour.
Thanks a lot Reza for your input. Regards Mohil On Mon, Apr 6, 2020 at 6:09 PM Reza Ardeshir Rokni <[email protected]> wrote: > Depends on the use case, Global state comes with the technical debt of > having to do your own GC, but comes with more control. You could > implement the pattern above with a long FixedWindow as well, which will > take care of the GC within the window bound. > > Sorry, its not a yes / no answer :-) > > On Tue, 7 Apr 2020 at 09:03, Mohil Khare <[email protected]> wrote: > >> Thanks a lot Reza for your quick response. Yeah saving the data in an >> external system after timer expiry makes sense. >> So do you suggest using a global window for maintaining state ? >> >> Thanks and regards >> Mohil >> >> On Mon, Apr 6, 2020 at 5:37 PM Reza Ardeshir Rokni <[email protected]> >> wrote: >> >>> Are you able to make use of the following pattern? >>> >>> Store StateA-metadata until no activity for Duration X, you can use a >>> Timer to check this, then expire the value, but store in an >>> external system. If you get a record that does want this value after >>> expiry, call out to the external system and store the value again in key >>> StateA-metadata. >>> >>> Cheers >>> >>> On Tue, 7 Apr 2020 at 08:03, Mohil Khare <[email protected]> wrote: >>> >>>> Hello all, >>>> We are attempting a implement a use case where beam (java sdk) reads >>>> two kind of records from data stream like Kafka: >>>> >>>> 1. Records of type A containing key and corresponding metadata. >>>> 2. Records of type B containing the same key, but no metadata. Beam >>>> then needs to fill metadata for records of type B by doing a lookup for >>>> metadata using keys received in records of type A. >>>> >>>> Idea is to save metadata or rather state for keys received in records >>>> of type A and then do a lookup when records of type B are received. >>>> I have implemented this using the "@State" construct of beam. However >>>> my problem is that we don't know when keys should expire. I don't think >>>> keeping a global window will be a good idea as there could be many keys >>>> (may be millions over a period of time) to be saved in a state. >>>> >>>> What is the best way to achieve this? I was reading about RedisIO, but >>>> found that it is still in the experimental stage. Can someone please >>>> recommend any solution to achieve this. >>>> >>>> Thanks and regards >>>> Mohil >>>> >>>> >>>> >>>> >>>> >>>>
