OK. Thanks TD. Does stateSnapshots() bring the snapshot of the state of all the keys managed by mapWithState or does it just bring the state of the keys in the current micro batch? Its kind of conflicting because the following link says that it brings the state only for the keys seen in the current batch. But, the code documentation says that it brings the state for all the keys. Also stateSnapshots() does it give the expired key state for the last time if we set it?
wordStream.mapWithState(stateSpec).stateSnapshots() https://docs.cloud.databricks.com/docs/spark/1.6/examples/Streaming%20mapWithState.html On Thu, Jul 13, 2017 at 3:00 PM, Tathagata Das <tathagata.das1...@gmail.com> wrote: > Yes. It does. > > On that note, Spark 2.2 (released couple of days ago) adds > mapGroupsWithState in Structured Streaming. That is like mapWithState on > steroids. Just saying. :) > > On Thu, Jul 13, 2017 at 1:01 PM, SRK <swethakasire...@gmail.com> wrote: > >> Hi, >> >> Do we need to specify checkpointing for mapWithState just like we do for >> updateStateByKey? >> >> Thanks, >> Swetha >> >> >> >> -- >> View this message in context: http://apache-spark-user-list. >> 1001560.n3.nabble.com/Does-mapWithState-need-checkpointing- >> to-be-specified-in-Spark-Streaming-tp28858.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> >> >
