One option is to use a NoSQL data store, such as hbase, for the two actions to exchange status information. Write to data store in action 1 and read from action 2.
Cheers On Wed, Jan 13, 2016 at 2:20 AM, Kira <mennou...@gmail.com> wrote: > Hi, > > So i have an action on one RDD that is relatively long, let's call it ac1; > what i want to do is to execute another action (ac2) on the same RDD to see > the evolution of the first one (ac1); for this end i want to use an > accumulator and read it's value progressively to see the changes on it (on > the fly) while ac1 is always running. My problem is that the accumulator is > only updated once the ac1 has been finished, this is not helpful for me :/ > . > > I ve seen here > < > http://apache-spark-user-list.1001560.n3.nabble.com/Asynchronous-Broadcast-from-driver-to-workers-is-it-possible-td15758.html > > > what may seem like a solution for me but it doesn t work : "While Spark > already offers support for asynchronous reduce (collect data from workers, > while not interrupting execution of a parallel transformation) through > accumulator" > > Another post suggested to use SparkListner to do that. > > are these solutions correct ? if yes, give me a simple exemple ? > are there other solutions ? > > thank you. > Regards > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Concurrent-Read-of-Accumulator-s-Value-tp25957.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
