Thanks a lot for your quick response! Your suggestion however would never
work for our use case. Ours is a streaming system that must process 100
thousand messages per second and produce immediate results and it's simply
impossible to rerun the job.
Our job is a streaming job broken down into vari
Hi Vishal
If your scenario is to update the data in full every time. One idea is to
rerun the job every time. For example, you have an
`EnrichWithDatabaseAndWebSerivce` job, which is responsible for reading all
data from a data source every time, and then joins the data with DB and Web
services. E
Yes. You have explained my requirements exactly as they are. My operator
will talk to multiple databases and a couple of web services to enrich
incoming input streams. I cannot think of a way to use the async IO
operator. So I thought maybe convert these 7-10 calls into async calls and
chain the Fu
Hi Vishal
I want to understand your needs first. Your requirements are: After a
stateful operator receives a notification, it needs to traverse all the
data stored in the operator state, communicate with an external system
during the traversal process (maybe similar to join?). In order to improve
Hello,
My application has a stateful operator which leverages RocksDB to store a
large amount of state. It, along with other operators receive configuration
as a broadcast stream (KeyedBroadcastProcessFunction). The operator depends
upon another input stream that triggers some communication with ex
