Hi, I have a rather complicated situation thats raised an issue regarding consuming multiple data sources for processing. Unlike the use cases I've found, I have 3 sources of different formats. There's one 'main' stream A that does the processing, and 2 sources B and C that provide elements required for processing. All 3 sources come from Kinesis streams and are updated in real time, so using a broadcast variable or passing a file to each node won't work. Also, each data source is different so I won't be able to join the streams.
Source A is an event stream, source B is used for processing events, and source C us used for filtering events. State needs to be maintained for source A and source B. So my question is, how can I pull in data from multiple, disparate sources and feed it into a single stream? Thanks
