Are you able to make use of the following pattern? Store StateA-metadata until no activity for Duration X, you can use a Timer to check this, then expire the value, but store in an external system. If you get a record that does want this value after expiry, call out to the external system and store the value again in key StateA-metadata.
Cheers On Tue, 7 Apr 2020 at 08:03, Mohil Khare <[email protected]> wrote: > Hello all, > We are attempting a implement a use case where beam (java sdk) reads two > kind of records from data stream like Kafka: > > 1. Records of type A containing key and corresponding metadata. > 2. Records of type B containing the same key, but no metadata. Beam then > needs to fill metadata for records of type B by doing a lookup for > metadata using keys received in records of type A. > > Idea is to save metadata or rather state for keys received in records of > type A and then do a lookup when records of type B are received. > I have implemented this using the "@State" construct of beam. However my > problem is that we don't know when keys should expire. I don't think > keeping a global window will be a good idea as there could be many keys > (may be millions over a period of time) to be saved in a state. > > What is the best way to achieve this? I was reading about RedisIO, but > found that it is still in the experimental stage. Can someone please > recommend any solution to achieve this. > > Thanks and regards > Mohil > > > > > >
