To your second question, that is what the 'invFunc' in reduceByKeyAndWindow() does. If you can supply an "un-reduce" function the windows can be updated rather than recomputed each time.
On Mon, Jun 9, 2014 at 5:39 AM, Yingjun Wu <wu.yj0...@gmail.com> wrote: > Dear all, > > I just run the window processing job using Spark-Streaming, and I have two > questions. First, how can I measure the latency of Spark-Streaming? Is there > any APIs that I can call directly? Second, is it true that the latency of > Spark-Streaming grows linearly with the window size? It seems that the > computation model of Spark-Streaming does not directly support incremental > computation. For example, to calculate the sum of five continuous numbers > from 1 to infinite, the first computation should accumulate 1, 2, 3, 4, 5, > and the second should accumulate 2, 3, 4, 5, 6. That is, the result we > obtained from the first computation cannot directly be applied to the next > computation round, correct? > > Thanks for your kind attention to this message. > > Regards, > Yingjun > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-window-processing-tp7234.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
