Great! BTW if you get the time and wanted to contribute back to beam there is a nice section to record cool patterns:
https://beam.apache.org/documentation/patterns/overview/ This would make a great one! On Tue, 7 Apr 2020 at 09:12, Mohil Khare <[email protected]> wrote: > No ... that's a valid answer. Since I wanted to have a long window size > per key and since we can't use state with session windows, I am using a > sliding window for let's say 72 hrs which starts every hour. > > Thanks a lot Reza for your input. > > Regards > Mohil > > On Mon, Apr 6, 2020 at 6:09 PM Reza Ardeshir Rokni <[email protected]> > wrote: > >> Depends on the use case, Global state comes with the technical debt of >> having to do your own GC, but comes with more control. You could >> implement the pattern above with a long FixedWindow as well, which will >> take care of the GC within the window bound. >> >> Sorry, its not a yes / no answer :-) >> >> On Tue, 7 Apr 2020 at 09:03, Mohil Khare <[email protected]> wrote: >> >>> Thanks a lot Reza for your quick response. Yeah saving the data in an >>> external system after timer expiry makes sense. >>> So do you suggest using a global window for maintaining state ? >>> >>> Thanks and regards >>> Mohil >>> >>> On Mon, Apr 6, 2020 at 5:37 PM Reza Ardeshir Rokni <[email protected]> >>> wrote: >>> >>>> Are you able to make use of the following pattern? >>>> >>>> Store StateA-metadata until no activity for Duration X, you can use a >>>> Timer to check this, then expire the value, but store in an >>>> external system. If you get a record that does want this value after >>>> expiry, call out to the external system and store the value again in key >>>> StateA-metadata. >>>> >>>> Cheers >>>> >>>> On Tue, 7 Apr 2020 at 08:03, Mohil Khare <[email protected]> wrote: >>>> >>>>> Hello all, >>>>> We are attempting a implement a use case where beam (java sdk) reads >>>>> two kind of records from data stream like Kafka: >>>>> >>>>> 1. Records of type A containing key and corresponding metadata. >>>>> 2. Records of type B containing the same key, but no metadata. Beam >>>>> then needs to fill metadata for records of type B by doing a lookup for >>>>> metadata using keys received in records of type A. >>>>> >>>>> Idea is to save metadata or rather state for keys received in records >>>>> of type A and then do a lookup when records of type B are received. >>>>> I have implemented this using the "@State" construct of beam. However >>>>> my problem is that we don't know when keys should expire. I don't think >>>>> keeping a global window will be a good idea as there could be many keys >>>>> (may be millions over a period of time) to be saved in a state. >>>>> >>>>> What is the best way to achieve this? I was reading about RedisIO, but >>>>> found that it is still in the experimental stage. Can someone please >>>>> recommend any solution to achieve this. >>>>> >>>>> Thanks and regards >>>>> Mohil >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>
