shuffling using netty in spark streaming

2014-06-12 Thread onpoq l
Hi, 1. Does netty perform better than the basic method for shuffling? I found the latency caused by shuffling in a streaming job is not stable with the basic method. 2. However, after I turn on netty for shuffling, I can only see the results for the first two batches, and then no output at all.

RE: shuffling using netty in spark streaming

2014-06-12 Thread Shao, Saisai
related metrics in netty based shuffle, so you may not see the shuffle fetch related metrics in web portal. Thanks Jerry From: onpoq l [mailto:onpo...@gmail.com] Sent: Thursday, June 12, 2014 2:35 PM To: user@spark.apache.org Subject: shuffling using netty in spark streaming Hi, 1. Does netty