Gradual slow down of the Streaming job (getCallSite at DStream.scala:294)

Here is the code for my streaming job. ~~val sparkConf = new SparkConf().setAppName(SparkStreamingJob) sparkConf.set(spark.serializer, org.apache.spark.serializer.KryoSerializer)sparkConf.set(spark.default.parallelism,

Re: Gradual slow down of the Streaming job (getCallSite at DStream.scala:294)

Which version of Spark Streaming are you using. When the batch processing time increases to 15-20 seconds, could you compare the task times compared to the tasks time when the application is just launched? Basically is the increase from 6 seconds to 15-20 seconds is caused by increase in

Re: Gradual slow down of the Streaming job (getCallSite at DStream.scala:294)

I am running the job on 1.1.1. I will let the job run overnight and send you more info on computation vs GC time tomorrow. BTW, do you know what the stage description named getCallSite at DStream.scala:294 might mean? Thanks,RK On Tuesday, December 30, 2014 6:02 PM, Tathagata Das