Hi Tathagata, How can i find the batch size?
Thanks, Sourav On Wed, Feb 5, 2014 at 2:02 PM, Tathagata Das <[email protected]>wrote: > Hi Sourav, > > For number of records received per second, you could use something like > this to calculate number of records in each batch, and divide it by your > batch size. > > yourKafkaStream.foreachRDD(rdd => { > val count = rdd.count > println("Current rate = " + (count / batchSize) + " records / second") > }) > > For the delay metrics, take a look at the StreamingListener > interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>. > You can create a streaming listener object and use > streamingContext.addListener(object) to attach the listener. > > > > On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra < > [email protected]> wrote: > >> HI, >> >> We are currently evaluating spark streaming for our analytics application. >> >> It reads from Kafka, processes and then persists into cassandra >> >> As part of poc project, we need to see the message processing rate of >> spark i.e. end to end time taken. >> >> I looked into metrics but did not find any way how to capture this info. >> >> Is there any way to see these metrics? >> >> I am using spark 0.9.0 >> >> Thanks, >> -- >> >> Sourav Chandra >> >> Senior Software Engineer >> >> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · >> >> [email protected] >> >> o: +91 80 4121 8723 >> >> m: +91 988 699 3746 >> >> skype: sourav.chandra >> >> Livestream >> >> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd >> Block, Koramangala Industrial Area, >> >> Bangalore 560034 >> >> www.livestream.com >> > > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · [email protected] o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
