Also, could you give a screenshot of the streaming UI. Even better, could you run it on Spark 1.4 which has a new streaming UI and then use that for debugging/screenshot?
TD On Thu, Jun 18, 2015 at 3:05 AM, Akhil Das <ak...@sigmoidanalytics.com> wrote: > Which version of spark? and what is your data source? For some reason, > your processing delay is exceeding the batch duration. And its strange that > you are not seeing any scheduling delay. > > Thanks > Best Regards > > On Thu, Jun 18, 2015 at 7:29 AM, Mike Fang <chyfan...@gmail.com> wrote: > >> Hi, >> >> >> >> I have a spark streaming program running for ~ 25hrs. When I check the >> Streaming UI tab. I found the “Waiting batches” is 144. But the “scheduling >> delay” is 0. I am a bit confused. >> >> If the “waiting batches” is 144, that means many batches are waiting in >> the queue to be processed? If this is the case, the scheduling delay should >> be high rather than 0. Am I missing anything? >> >> >> >> Thanks, >> >> Mike >> > >
