I think you could apply this pattern for that: https://beam.apache.org/documentation/patterns/side-inputs/#slowly-updating-side-input-using-windowing
On Sun, 10 Jul 2022 at 18:19, Mark Striebeck <[email protected]> wrote: > Hi, > > We want to use Beam (Python) to stream live stock market data to our > internal systems. Every day before the market opens, we need to refresh > some data (e.g. current ticker symbols). Currently, we are holding this > data in a simple dictionary in the relevant Beam transformations (loading > it in setup). > > What is the best way to refresh this data when we have updated it? We > could just restart the whole pipeline, but there must be some way to > communicate with a streaming Beam pipeline. > > Any ideas or pointers are appreciated! > > Mark >
