Hi, I am using spark streaming along with kafka dstream. and running the application against standalone cluster
Spark version => https://github.com/apache/incubator-spark/tree/branch-0.9 It seems after transformation, when i o foreachRDD, its not working. code snippet is below : --------------------------------------------------------------- val ssc = new StreamingContext(...) val stream = KafkaUtils.createStream(...) val processedStream = stream.flatMap(...) val newStream = processedStream.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _ - _, Seconds(1), Seconds(1), 2) newStream.foreachRDD(rdd => { rdd.foreach({ case (value, count) => { println("##########################################") println("value --> " + value + " with count --> " + count) println("##########################################") } }) }) --------------------------------------------------------------- If I run the application locally (local instead of spark://), it is working Can you suggest what is going on here? -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · [email protected] o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
