Re: Efficiency with different approaches of aggregation in Flink

2018-04-19 Thread Fabian Hueske
Hi Teena, I'd go with approach 2. The performance difference shouldn't be significant compared to 1. but it is much easier to implement, IMO. Avoid approach 3. It will be much slower because you need at least one call to an external data store and more difficult to implement. Flink's

Re: Efficiency with different approaches of aggregation in Flink

2018-04-19 Thread Puneet Kinra
Hi Teena If you are proceeding with point 3, no doubt it will add some overhead but major significance is that you are persisting the state as per some key. so there will not be data loss in case of the job failure. On Thu, Apr 19, 2018 at 11:45 AM, Teena Kappen // BPRISE <

Efficiency with different approaches of aggregation in Flink

2018-04-19 Thread Teena Kappen // BPRISE
Hi, If I have to aggregate a value in a stream of records, which one of the below approaches will be the most/least efficient? 1. Using a Global Window to aggregate the value and emit the record when it reaches a particular threshold value. 2. Using a FlatMap with a State Variable which