GitHub user davies opened a pull request:
https://github.com/apache/spark/pull/9746
[SPARK-11583] [Core]MapStatus Using RoaringBitmap More Properly
This PR upgrade the version of RoaringBitmap to 0.5.10, to optimize the
memory layout, will be much smaller when most of blocks are empty.
This PR is based on #9661 (fix conflicts), see all of the comments at
https://github.com/apache/spark/pull/9661 .
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davies/spark roaring_mapstatus
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9746.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9746
----
commit 75c3209c4bba8902eb0a4c1649864106f015c39f
Author: Kent Yao <[email protected]>
Date: 2015-11-12T11:25:02Z
MapStatus Using RoaringBitmap More Properly
commit 8f3eb7d25d0ca4465a5dc50aca5954a444ef82e0
Author: Kent Yao <[email protected]>
Date: 2015-11-13T02:24:13Z
update roaringbitmap to 0.5.10 and runOptimize
commit b4b7a1c835235826d512f1993c00ca35e445fcba
Author: Kent Yao <[email protected]>
Date: 2015-11-13T08:38:29Z
add roaringbitmap runOptimise tests
commit b9cc41439203f6924e75836a39fc603ea0714ceb
Author: Kent Yao <[email protected]>
Date: 2015-11-16T03:05:51Z
runOptimize and trim only
commit 64e7295c7fd0f193e8b74f07a2823a478e6c1796
Author: Kent Yao <[email protected]>
Date: 2015-11-16T08:01:00Z
fix style
commit 1716a24cece9bc4d9d418da3023e4132aa2b8d65
Author: Davies Liu <[email protected]>
Date: 2015-11-16T23:05:41Z
Merge branch 'master' of github.com:apache/spark into roaring_mapstatus
Conflicts:
core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala
pom.xml
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
