Hi,

On Thu, Jan 29, 2015 at 9:52 AM, Tobias Pfeiffer <[email protected]> wrote:

> Hi,
>
> On Thu, Jan 29, 2015 at 1:54 AM, YaoPau <[email protected]> wrote:
>>
>> My thinking is to maintain state in an RDD and update it an persist it
>> with
>> each 2-second pass, but this also seems like it could get messy.  Any
>> thoughts or examples that might help me?
>>
>
> I have just implemented some timestamp-based windowing on DStreams (can't
> share the code now, but will be published a couple of months ahead),
> although with the assumption that items are in correct order. The main
> challenge (rather technical) was to keep proper state across RDD boundaries
> and to tell the state "you can mark this partial window from the last
> interval as 'complete' now" without shuffling too much data around. For
> example, if there are some empty intervals, you don't know when the next
> item to go into the partial window will arrive, or if there will be one at
> all. I guess if you want to have out-of-order tolerance, that will become
> even trickier, as you need to define and think about some timeout for
> partial windows in your state...
>

Sorry, it took ages to get the code published, and I am not involved in
that project any more and cannot provide a lot of support, but if you are
interested in the code, here it is:

https://github.com/jubatus/jubaql-server/blob/master/processor/src/main/scala/us/jubat/jubaql_server/processor/SlidingWindow.scala

The usage can be seen in

https://github.com/jubatus/jubaql-server/blob/master/processor/src/test/scala/us/jubat/jubaql_server/processor/SlidingStreamSpec.scala

and it basically boils down to

  val inputStream: DStream[(Long, T)] = ssc.queueStream(itemsQueue,
oneAtATime = true)
  val windowStream: DStream[(Long, (Long, T))] =
SlidingWindow.byTimestamp(inputStream,
length, step)

where the first Long in the result is a window ID and the second Long is
the original timestamp, so now you can use any of the groupByKey,
reduceByKey etc. functions.

Hope this helps anyone,
Tobias

Reply via email to