Hi Sourav, >From "foreach not working" you mean Job is not getting scheduled at batch interval? I came across similar issue with standalone mode. You can try increasing your batch interval.
I increase the priority of RecurringTimer (incubator-spark/streaming/src/main/scala/org/apache/spark/streamingutil/RecurringTimer.scala) thread to get it working Regards Anita On 23 January 2014 14:31, Sourav Chandra <[email protected]>wrote: > Hi, > > I am using spark streaming along with kafka dstream. and running the > application against standalone cluster > > Spark version => https://github.com/apache/incubator-spark/tree/branch-0.9 > > It seems after transformation, when i o foreachRDD, its not working. > > code snippet is below : > --------------------------------------------------------------- > val ssc = new StreamingContext(...) > val stream = KafkaUtils.createStream(...) > val processedStream = stream.flatMap(...) > val newStream = processedStream.map(x => (x, 1L)).reduceByKeyAndWindow(_ + > _, _ - _, Seconds(1), Seconds(1), 2) > newStream.foreachRDD(rdd => { > rdd.foreach({ > case (value, count) => { > println("##########################################") > println("value --> " + value + " with count --> " + count) > println("##########################################") > } > }) > }) > > --------------------------------------------------------------- > > If I run the application locally (local instead of spark://), it is working > > Can you suggest what is going on here? > > -- > > Sourav Chandra > > Senior Software Engineer > > · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · > > [email protected] > > o: +91 80 4121 8723 > > m: +91 988 699 3746 > > skype: sourav.chandra > > Livestream > > "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd > Block, Koramangala Industrial Area, > > Bangalore 560034 > > www.livestream.com >
