Yes! Sorry for the confusion but batchInterval is a more appropriate term. It is the same batch interval/duration that you provide when you are creating the StreamingContext.
TD On Wed, Feb 5, 2014 at 2:44 AM, Pankaj Mittal <[email protected]>wrote: > Isn't it println("Current rate = " + (count / *batchInterval*) + " > records / second") ? > > > > > On Wed, Feb 5, 2014 at 2:05 PM, Sourav Chandra < > [email protected]> wrote: > >> Hi Tathagata, >> >> How can i find the batch size? >> >> Thanks, >> Sourav >> >> >> >> >> On Wed, Feb 5, 2014 at 2:02 PM, Tathagata Das < >> [email protected]> wrote: >> >>> Hi Sourav, >>> >>> For number of records received per second, you could use something like >>> this to calculate number of records in each batch, and divide it by your >>> batch size. >>> >>> yourKafkaStream.foreachRDD(rdd => { >>> val count = rdd.count >>> println("Current rate = " + (count / batchSize) + " records / second") >>> }) >>> >>> For the delay metrics, take a look at the StreamingListener >>> interface<http://spark.incubator.apache.org/docs/latest/api/streaming/index.html#org.apache.spark.streaming.scheduler.StreamingListener>. >>> You can create a streaming listener object and use >>> streamingContext.addListener(object) to attach the listener. >>> >>> >>> >>> On Tue, Feb 4, 2014 at 10:32 PM, Sourav Chandra < >>> [email protected]> wrote: >>> >>>> HI, >>>> >>>> We are currently evaluating spark streaming for our analytics >>>> application. >>>> >>>> It reads from Kafka, processes and then persists into cassandra >>>> >>>> As part of poc project, we need to see the message processing rate of >>>> spark i.e. end to end time taken. >>>> >>>> I looked into metrics but did not find any way how to capture this info. >>>> >>>> Is there any way to see these metrics? >>>> >>>> I am using spark 0.9.0 >>>> >>>> Thanks, >>>> -- >>>> >>>> Sourav Chandra >>>> >>>> Senior Software Engineer >>>> >>>> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · >>>> >>>> [email protected] >>>> >>>> o: +91 80 4121 8723 >>>> >>>> m: +91 988 699 3746 >>>> >>>> skype: sourav.chandra >>>> >>>> Livestream >>>> >>>> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd >>>> Block, Koramangala Industrial Area, >>>> >>>> Bangalore 560034 >>>> >>>> www.livestream.com >>>> >>> >>> >> >> >> -- >> >> Sourav Chandra >> >> Senior Software Engineer >> >> · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · >> >> [email protected] >> >> o: +91 80 4121 8723 >> >> m: +91 988 699 3746 >> >> skype: sourav.chandra >> >> Livestream >> >> "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd >> Block, Koramangala Industrial Area, >> >> Bangalore 560034 >> >> www.livestream.com >> > >
