Hello all, We are attempting a implement a use case where beam (java sdk) reads two kind of records from data stream like Kafka:
1. Records of type A containing key and corresponding metadata. 2. Records of type B containing the same key, but no metadata. Beam then needs to fill metadata for records of type B by doing a lookup for metadata using keys received in records of type A. Idea is to save metadata or rather state for keys received in records of type A and then do a lookup when records of type B are received. I have implemented this using the "@State" construct of beam. However my problem is that we don't know when keys should expire. I don't think keeping a global window will be a good idea as there could be many keys (may be millions over a period of time) to be saved in a state. What is the best way to achieve this? I was reading about RedisIO, but found that it is still in the experimental stage. Can someone please recommend any solution to achieve this. Thanks and regards Mohil
