Github user lemire commented on a diff in the pull request:
https://github.com/apache/spark/pull/9661#discussion_r44728872
--- Diff: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ---
@@ -176,15 +179,17 @@ private[spark] object HighlyCompressedMapStatus {
// From a compression standpoint, it shouldn't matter whether we track
empty or non-empty
--- End diff --
Yes, {{RoaringBitmap}} is not symmetric with respect to flips, but that is
true also of {{BitSet}}. In Java's BitSet, if you do a flip, can change the
{{wordsInUse}} and if you do call trimToSize() afterward, the memory usage
would change. What {{runOptimize}} will buy you however is that if you have a
small memory usage before the flip (compared to a BitSet), your memory usage
will still be "small".
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
