GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/3378
[SPARK-4505][Core] Add a ClassTag parameter to CompactBuffer[T]
Added a ClassTag parameter to CompactBuffer. So CompactBuffer[T] can create
primitive arrays for primitive types. It will reduce the memory usage for
primitive types significantly and only pay minor performance lost.
Here is my test code:
```Scala
// Call org.apache.spark.util.SizeEstimator.estimate
def estimateSize(obj: AnyRef): Long = {
val c = Class.forName("org.apache.spark.util.SizeEstimator$")
val f = c.getField("MODULE$")
val o = f.get(c)
val m = c.getMethod("estimate", classOf[Object])
m.setAccessible(true)
m.invoke(o, obj).asInstanceOf[Long]
}
sc.parallelize(1 to 10000).groupBy(_ => 1).foreach {
case (k, v) =>
println(v.getClass() + " size: " + estimateSize(v))
}
```
Using the previous CompactBuffer outputed
```
class org.apache.spark.util.collection.CompactBuffer size: 313358
```
Using the new CompactBuffer outputed
```
class org.apache.spark.util.collection.CompactBuffer size: 65712
```
In this case, the new `CompactBuffer` only used 20% memory of the previous
one. It's really helpful for `groupByKey` when using a primitive value.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-4505
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/3378.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3378
----
commit 4abdbba566aa776280cc20689fa901e17634e5fe
Author: zsxwing <[email protected]>
Date: 2014-11-20T01:35:27Z
Add a ClassTag parameter to reduce the memory usage of CompactBuffer[T]
when T is a primitive type
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
