Hi Teena If you are proceeding with point 3, no doubt it will add some overhead but major significance is that you are persisting the state as per some key. so there will not be data loss in case of the job failure.
On Thu, Apr 19, 2018 at 11:45 AM, Teena Kappen // BPRISE < teena.kap...@bprise.com> wrote: > Hi, > > > > If I have to aggregate a value in a stream of records, which one of the > below approaches will be the most/least efficient? > > > > 1. Using a Global Window to aggregate the value and emit the record > when it reaches a particular threshold value. > 2. Using a FlatMap with a State Variable which gets updated with each > incoming record and emit the record when it reaches the threshold value. > 3. Using a FlatMap to store the aggregated value in an in-memory DB > like Redis and query the value and update it with each incoming record, and > emit the record when it reaches the threshold value. > > > > Please rate the three approaches according to their efficiency. > > > > Regards, > > Teena > -- *Cheers * *Puneet Kinra* *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com <puneet.ki...@customercentria.com>* *e-mail :puneet.ki...@customercentria.com <puneet.ki...@customercentria.com>*
