-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3175/
-----------------------------------------------------------

Review request for giraph.


Summary
-------

Currently, the amount of memory that Giraph uses for messaging is huge. This 
JIRA will reduce the messaging memory by half and provide periodic updates of 
memory for debugging. Details are below:

Refactored RandomMessageBenchmark to an internal vertex class. Added 
aggregators to RandomMessagesBenchmark to track bytes, messages, and time for 
the messaging. Adjusted the postSuperstep() to be called after the flush() for 
more accurate timings.

Added periodic minute updates for message flushing (which can take a while, 
especially on the memory benchmark). This helps to see how progress is going 
and gives an ETA.

Memory optimizations include:

-Clear the message list after computation
-Free vertex messages on the source as the flush is going on
-TreeMap -> HashMap for VertexMutations
-Sizing the ArrayList properly in transientInMessages


This addresses bug GIRAPH-104.
    https://issues.apache.org/jira/browse/GIRAPH-104


Diffs
-----

  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/LongSumAggregator.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GraphMapper.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerContext.java
 1213849 
  
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/utils/MemoryUtils.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/3175/diff


Testing
-------

Passed local and Hadoop unittests.  RandomMessageBenchmark was run at scale on 
a real cluster.


Thanks,

Avery

Reply via email to