On Mon, Oct 2, 2023 at 2:00 AM Jan Lukavský <je...@seznam.cz> wrote:
> Hi, > > this depends on how exactly you plan to calculate the average. The > original definition is based on exponentially decreasing weight of more > distant (older if time is on the x-axis) data points. This (technically) > means that this average at any point X1 depends on all values X0 <= X1. > This would therefore require buffering (using GroupByKey) all elements in > global window, doing the sorting manually and then computing the new value > of the average triggering after each element. This is probably the > technically correct, but most computationally intensive variant. > To clarify - you would probably buffer the elements in OrderedListState, and set periodic event-time timers to fetch them and compute the average. OrderedListState will return the elements in order, so you wouldn't have to sort. This is assuming you are talking about streaming pipelines. > If the average is done over time intervals, then an other option could be > to define a cut-off interval T, i.e. set the exponentially vanishing weight > of value of data points to be zero at some T0 < T1 - T. If the data points > come at some discrete time-intervals (minutes, hours, days), then this > could mean you can split the data into time sliding windows (window > interval being the cut-off interval, and slide the update interval) and > assign weight for each data point in the particular time interval - i.e. > how much weight does the data point have at the time of end of the sliding > window. With this you could then using CombineFn to count and sum the > weighted averages, which would be much more efficient. > > Best, > > Jan > On 9/30/23 17:08, Balogh, György wrote: > > Hi, > I want to calculate the exponential moving average of a signal using beam > in java. > I understand there is no time order guarantee on incoming data. What would > be the simplest solution for this? > Thank you, > > -- > > György Balogh > CTO > E gyorgy.bal...@ultinous.com <zsolt.sala...@ultinous.com> > M +36 30 270 8342 <+36%2030%20270%208342> > A HU, 1117 Budapest, Budafoki út 209. > W www.ultinous.com > >
