Github user aokolnychyi commented on a diff in the pull request:
https://github.com/apache/spark/pull/16024#discussion_r89788527
--- Diff: docs/programming-guide.md ---
@@ -1424,29 +1431,38 @@ accum.value();
// returns 10
{% endhighlight %}
-Programmers can also create their own types by subclassing
-[AccumulatorParam](api/java/index.html?org/apache/spark/AccumulatorParam.html).
-The AccumulatorParam interface has two methods: `zero` for providing a
"zero value" for your data
-type, and `addInPlace` for adding two values together. For example,
supposing we had a `Vector` class
-representing mathematical vectors, we could write:
+While this code used the built-in support for accumulators of type Long,
programmers can also
+create their own types by subclassing
[AccumulatorV2](api/scala/index.html#org.apache.spark.util.AccumulatorV2).
+The AccumulatorV2 abstract class has several methods which one has to
override:
+`reset` for resetting the accumulator to zero, `add` for adding another
value into the accumulator,
+and `merge` for merging another same-type accumulator into this one. Other
methods that must be overridden
+are contained in the [API
documentation](api/scala/index.html#org.apache.spark.util.AccumulatorV2).
+For example, supposing we had a `MathVector` class representing
mathematical vectors, we could write:
{% highlight java %}
-class VectorAccumulatorParam implements AccumulatorParam<Vector> {
- public Vector zero(Vector initialValue) {
- return Vector.zeros(initialValue.size());
+class MathVectorAccumulatorV2 implements AccumulatorV2<MathVector,
MathVector> {
+
+ private MathVector mathVector = MathVector.newZeroVector();
+
+ public void reset() {
+ mathVector.reset();
}
- public Vector addInPlace(Vector v1, Vector v2) {
- v1.addInPlace(v2); return v1;
+
+ public void add(MathVector v) {
+ mathVector.add(v);
}
+
+ ...
}
-// Then, create an Accumulator of this type:
-Accumulator<Vector> vecAccum = sc.accumulator(new Vector(...), new
VectorAccumulatorParam());
+// Create an instance
+MathVectorAccumulatorV2 mathVectorAcc = new MathVectorAccumulatorV2();
+// Register it
+jsc.sc().register(mathVectorAcc, "MathVectorAcc1");
{% endhighlight %}
-In Java, Spark also supports the more general
[Accumulable](api/java/index.html?org/apache/spark/Accumulable.html)
-interface to accumulate data where the resulting type is not the same as
the elements added (e.g. build
-a list by collecting together elements).
+Note that, when programmers define their own type of AccumulatorV2,the
resulting type can be different
--- End diff --
Sure, I'll update that. Thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
