Re: using updating shared data

That is not fully correct. While in practice it may not matter, ignoring the timestamp of control messages may result in non-deterministic behavior, as during a restart the control message may be processed in a different order in relation to the other stream. So the output of multiple runs may

Re: using updating shared data

Sounds like a good idea. because in the control stream the time doesn't really matters. Thanks !!! On Fri, Jan 4, 2019 at 11:13 AM David Anderson wrote: > Another solution to the watermarking issue is to write an > AssignerWithPeriodicWatermarks for the control stream that always returns >

Re: using updating shared data

Another solution to the watermarking issue is to write an AssignerWithPeriodicWatermarks for the control stream that always returns Watermark.MAX_WATERMARK as the current watermark. This produces watermarks for the control stream that will effectively be ignored. On Thu, Jan 3, 2019 at 9:18 PM

Re: using updating shared data

Thanks for the tip Elias! On Wed, Jan 2, 2019 at 9:44 PM Elias Levy wrote: > One thing you must be careful of, is that if you are using event time > processing, assuming that the control stream will only receive messages > sporadically, is that event time will stop moving forward in the

Re: using updating shared data

One thing you must be careful of, is that if you are using event time processing, assuming that the control stream will only receive messages sporadically, is that event time will stop moving forward in the operator joining the streams while the control stream is idle. You can get around this by

Re: using updating shared data

Yes exactly Avi. Cheers, Till On Wed, Jan 2, 2019 at 5:42 PM Avi Levi wrote: > Thanks Till I will defiantly going to check it. just to make sure that I > got you correctly. you are suggesting the the list that I want to broadcast > will be broadcasted via control stream and it will be than be

Re: using updating shared data

Thanks Till I will defiantly going to check it. just to make sure that I got you correctly. you are suggesting the the list that I want to broadcast will be broadcasted via control stream and it will be than be kept in the relevant operator state correct ? and updates (CRUD) on that list will be

Re: using updating shared data

Hi Avi, you could use Flink's broadcast state pattern [1]. You would need to use the DataStream API but it allows you to have two streams (input and control stream) where the control stream is broadcasted to all sub tasks. So by ingesting messages into the control stream you can send model

Re: using updating shared data

Im trying to understand your use case. What is the source of the data ? FS ,KAFKA else ? On Tue, Jan 1, 2019 at 6:29 PM Avi Levi wrote: > Hi, > I have a list (couple of thousands text lines) that I need to use in my > map function. I read this article about broadcasting variables >

using updating shared data

Hi, I have a list (couple of thousands text lines) that I need to use in my map function. I read this article about broadcasting variables or using distributed cache