Re: Data exchange between tasks (operators/sources) at streaming api runtime

Yes, makes sense, I think consider one of those better options. Thanks! Ishwara Sent from my iPhone > On Jan 25, 2018, at 7:12 AM, Piotr Nowojski wrote: > > If you want to go this way, you could: > - as you proposed use some busy waiting with reading some file from a

Re: Data exchange between tasks (operators/sources) at streaming api runtime

If you want to go this way, you could: - as you proposed use some busy waiting with reading some file from a distributed file system - wait for some network message (opening your own socket) - use some other external system for this purpose: Kafka? Zookeeper? Although all of them seems hacky

Re: Data exchange between tasks (operators/sources) at streaming api runtime

The FLIP-17 is promising. Until it’s available I’m planning to do this: extend Kafka consumer and add logic to hold consuming until other source (fixed set) completes sending and those messages are processed by the application. However the question is to how to let the Kafka consumer know that

Re: Data exchange between tasks (operators/sources) at streaming api runtime

Hi, As far as I know there is currently no simple way to do this: Join stream with static data in https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API and

Data exchange between tasks (operators/sources) at streaming api runtime

Hello, I have a scenario where I've two sources, one of them is source of fixed list of ids for preloading (caching certain info which is slow) and second one is the kafka consumer. I need to run Kafka after first one completes. I need a mechanism to let the Kafka consumer know that it can start