Hi,
How to have a single reference of a class across all the executors in Spark
Streaming? The contents of the class will be updated at all the executors.
Would using it as a variable inside updateStateByKey guarantee that
reference is updated across all the executors and no
concurrentModificationException? Following is how I am trying to use a
Tracker Class across all the JVMs.
val trackerClass = new TrackerClass();
val runningCounts = pairs.updateStateByKey[Int](updateFunction _)
def updateFunction(newValues: Seq[Int], runningCount: Option[Int]):
Option[Int] = {
getMergedSession(this.trackerClass)
Some(newCount)
}
Thanks,
Swetha
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-have-Single-refernce-of-a-class-in-Spark-Streaming-tp25103.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
