@Mayur, thanks for the excellent explanation. :) @Chris, yes, after second thought, I think we can achieve the sliding window implementation with existing code base. Maybe adding a simple example in hello-samza will be helpful since the implementation is not very explicit but sliding window is a common use case.
Fang, Yan [email protected] +1 (206) 849-4108 On Wed, Sep 10, 2014 at 9:39 AM, Chris Riccomini < [email protected]> wrote: > Hey Guys, > > Our thought was that sliding window could be implemented with a buffer > inside of a process() call. For example, you might have a list of 10 > elements, and every time process() is invoked, you could add the message > to the ahead of the list, and dequeue the last element from the buffer (if > it already has 10 elements in it). > > So, Samza currently doesn't support any explicit sliding window, but it > seems to me that you could implement it in StreamTask.process() if you > need to. > > Cheers, > Chris > > On 9/9/14 10:31 PM, "Mayur Rustagi" <[email protected]> wrote: > > >Slide window is another dimension to processing > > > >say my batch is 3 sec & window is 9 sec then this is what I get > > > >[ x1 x2 x3][x4 x5 x6][x7 x8 x9] > > > >This is using slide but slide is equal to window size, but i may want to > >get last 3 elements at any point of time then that would be > > > > > >[ x1 x2 x3] after3sec [ x2 x3 x4] after3sec [ x3 x4 x5] after3sec [ > >x4 x5 x6] > > > >to implement this you use batch of 3 sec, window of 9 sec & slide > >duration of 3 sec. So we are sliding every 3 sec & also getting a batch > >every 3 sec. > > > > > > > > > > > >-- > >Regards, > >Mayur Rustagi > >Ph: +1 (760) 203 3257 > >http://www.sigmoidanalytics.com > >@mayur_rustagi > > > >On Wed, Sep 10, 2014 at 5:25 AM, Yan Fang <[email protected]> wrote: > > > >> Hi guys, > >> realize that both Storm and Spark Streaming have sliding window > >> implementation while Samza only has the fixed window (not sure if it's a > >> correct name). I think you guys must consider this idea at the > >>beginning of > >> designing the Samza. What was the thought? Thank you. > >> Cheers, > >> Fang, Yan > >> [email protected] > >> +1 (206) 849-4108 > >
