What kind of window functions are we talking about? Structured streaming only supports time window aggregates, not the more general sql window function (sum(x) over (partition by ... order by ...)) aggregates.
The basic idea is that you use incremental aggregation and store the aggregation buffer (not the end result) in a state store after each increment. When an new batch comes in, you perform aggregation on that batch, merge the result of that aggregation with the buffer in the state store, update the state store and return the new result. This is much harder than it sounds, because you need to maintain state in a fault tolerant way and you need to have some eviction policy (watermarks for instance) for aggregation buffers to prevent the state store from reaching an infinite size. On Thu, Nov 17, 2016 at 12:19 AM, assaf.mendelson <assaf.mendel...@rsa.com> wrote: > Hi, > > I have been trying to figure out how structured streaming handles window > functions efficiently. > > The portion I understand is that whenever new data arrived, it is grouped > by the time and the aggregated data is added to the state. > > However, unlike operations like sum etc. window functions need the > original data and can change when data arrives late. > > So if I understand correctly, this would mean that we would have to save > the original data and rerun on it to calculate the window function every > time new data arrives. > > Is this correct? Are there ways to go around this issue? > > > > Assaf. > > ------------------------------ > View this message in context: structured streaming and window functions > <http://apache-spark-developers-list.1001551.n3.nabble.com/structured-streaming-and-window-functions-tp19930.html> > Sent from the Apache Spark Developers List mailing list archive > <http://apache-spark-developers-list.1001551.n3.nabble.com/> at > Nabble.com. >
