Re: State management in spark-streaming
Well, the trackStateByKey has been renamed to mapWithState in upcoming 1.6. And regarding the usecase, you can easily implement this with updateStateByKey. See https://github.com/apache/spark/blob/branch-1.5/examples/src/main/scala/org/apache/spark/examples/streaming/StatefulNetworkWordCount.scala On Thu, Dec 10, 2015 at 7:46 PM, manasdebashiskar <poorinsp...@gmail.com> wrote: > Have you taken a look at trackStateBykey in spark streaming (coming in > spark > 1.6) > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/State-management-in-spark-streaming-tp25608p25681.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: State management in spark-streaming
Have you taken a look at trackStateBykey in spark streaming (coming in spark 1.6) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/State-management-in-spark-streaming-tp25608p25681.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
State management in spark-streaming
Hi, I have spark streaming with mqtt as my source. There are continuous events of flame sensors i.e. Fire and no Fire. I want to generate Fire event when the newly event is for Fire and want to ignore all subsequent event until No fire event is happened. Similarly If i get No-Fire Event i will ignore all subsequent No-Fire events and will wait for Fire Events. Is it possible to handle this logic in custom receiver? If yes how can we remember last state i.e. Fire/No-Fire in receiver. Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/State-management-in-spark-streaming-tp25608.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
