Re: Maintaining overall cumulative data in Spark Streaming

2015-10-30 Thread Silvio Fiorito
ber 30, 2015 at 9:29 AM To: skaarthik oss mailto:skaarthik@gmail.com>> Cc: dev mailto:d...@spark.apache.org>>, user mailto:user@spark.apache.org>> Subject: Re: Maintaining overall cumulative data in Spark Streaming How to we reset the aggregated statistics to null? Regards,

Re: Maintaining overall cumulative data in Spark Streaming

2015-10-30 Thread Sandeep Giri
How to we reset the aggregated statistics to null? Regards, Sandeep Giri, +1 347 781 4573 (US) +91-953-899-8962 (IN) www.KnowBigData.com. Phone: +1-253-397-1945 (Office) [image: linkedin icon] [image: other site icon]

RE: Maintaining overall cumulative data in Spark Streaming

2015-10-29 Thread Sandeep Giri
Yes, update state by key worked. Though there are some more complications. On Oct 30, 2015 8:27 AM, "skaarthik oss" wrote: > Did you consider UpdateStateByKey operation? > > > > *From:* Sandeep Giri [mailto:sand...@knowbigdata.com] > *Sent:* Thursday, October 29, 2015 3:09 PM > *To:* user ; dev

RE: Maintaining overall cumulative data in Spark Streaming

2015-10-29 Thread Silvio Fiorito
You could use updateStateByKey. There's a stateful word count example on Github. https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala From: Sandeep Giri