Yes, please tell us what operation are you using. TD
On Fri, Jun 19, 2015 at 11:42 AM, Cody Koeninger <c...@koeninger.org> wrote: > Is there any more info you can provide / relevant code? > > On Fri, Jun 19, 2015 at 1:23 PM, Tim Smith <secs...@gmail.com> wrote: > >> Update on performance of the new API: the new code using the >> createDirectStream API ran overnight and when I checked the app state in >> the morning, there were massive scheduling delays :( >> >> Not sure why and haven't investigated a whole lot. For now, switched back >> to the createStream API build of my app. Yes, for the record, this is with >> CDH 5.4.1 and Spark 1.3. >> >> >> >> On Thu, Jun 18, 2015 at 7:05 PM, Tim Smith <secs...@gmail.com> wrote: >> >>> Thanks for the super-fast response, TD :) >>> >>> I will now go bug my hadoop vendor to upgrade from 1.3 to 1.4. Cloudera, >>> are you listening? :D >>> >>> >>> >>> >>> >>> On Thu, Jun 18, 2015 at 7:02 PM, Tathagata Das < >>> tathagata.das1...@gmail.com> wrote: >>> >>>> Are you using Spark 1.3.x ? That explains. This issue has been fixed in >>>> Spark 1.4.0. Bonus you get a fancy new streaming UI with more awesome >>>> stats. :) >>>> >>>> On Thu, Jun 18, 2015 at 7:01 PM, Tim Smith <secs...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I just switched from "createStream" to the "createDirectStream" API >>>>> for kafka and while things otherwise seem happy, the first thing I noticed >>>>> is that stream/receiver stats are gone from the Spark UI :( Those stats >>>>> were very handy for keeping an eye on health of the app. >>>>> >>>>> What's the best way to re-create those in the Spark UI? Maintain >>>>> Accumulators? Would be really nice to get back receiver-like stats even >>>>> though I understand that "createDirectStream" is a receiver-less design. >>>>> >>>>> Thanks, >>>>> >>>>> Tim >>>>> >>>>> >>>>> >>>> >>> >> >
