Save half of maximum memory used from messaging
Issue Type: Improvement
Reporter: Avery Ching
Currently, the amount of memory that Giraph uses for messaging is huge. This
JIRA will reduce the messaging memory by half and provide periodic updates of
memory for debugging. Details are below:
Refactored RandomMessageBenchmark to an internal vertex class. Added
aggregators to RandomMessagesBenchmark to track bytes, messages, and time for
the messaging. Adjusted the postSuperstep() to be called after the flush() for
more accurate timings.
Added periodic minute updates for message flushing (which can take a while,
especially on the memory benchmark). This helps to see how progress is going
and gives an ETA.
Memory optimizations include:
- Clear the message list after computation
- Free vertex messages on the source as the flush is going on
- TreeMap -> HashMap for VertexMutations
- Sizing the ArrayList properly in transientInMessages
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira