Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/11105#discussion_r86381839
--- Diff: core/src/main/scala/org/apache/spark/util/AccumulatorV2.scala ---
@@ -42,18 +60,43 @@ private[spark] case class AccumulatorMetadata(
* `OUT` should be a type that can be read atomically (e.g., Int, Long),
or thread-safely
* (e.g., synchronized collections) because it will be read from other
threads.
*/
-abstract class AccumulatorV2[IN, OUT] extends Serializable {
+abstract class AccumulatorV2[@specialized(Int, Long, Double) IN, OUT]
extends Serializable {
--- End diff --
hmm, it doesn't look this way to me. If I look at the compiled code for
AccumulatorV2 in the spark2 binaries, there are no specialized methods. It
looks like boxing does happen.
I do see test failures in `AccumulatorV2Suite` without this change, but
looks like that can be fixed by just tweaking the specialized versions of these
functions that are manually defined in `DoubleAccumulator` and
`LongAccumulator`, eg.
```scala
/**
* Adds v to the accumulator, i.e. increment sum by v and count by 1.
* @since 2.1.0
*/
def add(v: Double): Unit = {
if (metadata != null && metadata.dataProperty) {
dataPropertyAdd(v: jl.Double)
} else {
addImpl(v)
}
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
