Re: Issue while trying to aggregate with a sliding window

2014-06-17 Thread onpoq l
There is a bug: https://github.com/apache/spark/pull/961#issuecomment-45125185 On Tue, Jun 17, 2014 at 8:19 PM, Hatch M hatchman1...@gmail.com wrote: Trying to aggregate over a sliding window, playing with the slide duration. Playing around with the slide interval I can see the aggregation

shuffling using netty in spark streaming

2014-06-12 Thread onpoq l
Hi, 1. Does netty perform better than the basic method for shuffling? I found the latency caused by shuffling in a streaming job is not stable with the basic method. 2. However, after I turn on netty for shuffling, I can only see the results for the first two batches, and then no output at all.