Btw, here is a great article about accumulators and all their related
traps!
http://imranrashid.com/posts/Spark-Accumulators/ (I'm not the author)
On 16 March 2016 at 18:24, swetha kasireddy
wrote:
> OK. I did take a look at them. So once I have an accumulater for a
>
OK. I did take a look at them. So once I have an accumulater for a HashSet,
how can I check if a particular key is already present in the HashSet
accumulator? I don't see any .contains method there. My requirement is that
I need to keep accumulating the keys in the HashSet across all the tasks in
Have you looked at using Accumulable interface, Take a look at Spark
documentation at
http://spark.apache.org/docs/latest/programming-guide.html#accumulators it
gives example of how to use vector type for accumalator, which might be very
close to what you need
--
View this message in context:
Please take a look at:
core/src/test/scala/org/apache/spark/AccumulatorSuite.scala
FYI
On Tue, Mar 15, 2016 at 4:29 PM, SRK wrote:
> Hi,
>
> How do I add an accumulator for a Set in Spark?
>
> Thanks!
>
>
>
> --
> View this message in context:
>
Hi,
How do I add an accumulator for a Set in Spark?
Thanks!
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-add-an-accumulator-for-a-Set-in-Spark-tp26510.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.