[ 
https://issues.apache.org/jira/browse/GIRAPH-104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168938#comment-13168938
 ] 

Avery Ching commented on GIRAPH-104:
------------------------------------

By the way, here's example output from the changes to RandomMessageBenchmark.  
It will help us qualify messaging improvements.

2011-12-12 23:58:54,887 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Outputing statistics for superstep 4
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 superstep total bytes sent : 60000000000
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 total bytes sent : 240000000000
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 superstep total messages : 6000000
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 total messages : 24000000
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 superstep total millis : 854309
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 total millis : 3718123
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 workers : 5
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Superstep megabytes / second = 334.8932235547969
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Total megabytes / second = 307.7921789267058
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Superstep messages / second = 35116.09967821947
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Total messages / second = 32274.349181024943
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Superstep megaabytes / second / worker = 66.97864471095939
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Total megabytes / second / worker = 61.55843578534116
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Superstep messages / second / worker = 7023.219935643894
2011-12-12 23:58:54,888 INFO 
org.apache.giraph.benchmark.RandomMessageBenchmark$RandomMessageBenchmarkWorkerContext:
 Total messages / second / worker = 6454.869836204989
2011-12-12 23:58:57,627 INFO org.apache.giraph.comm.BasicRPCCommunications: 
flush: starting for superstep 4 totalMem = 20463.375M, maxMem = 20463.375M, 
freeMem = 6571.4233M

                
> Save half of maximum memory used from messaging
> -----------------------------------------------
>
>                 Key: GIRAPH-104
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-104
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Avery Ching
>            Assignee: Avery Ching
>            Priority: Critical
>         Attachments: GIRAPH-104.diff
>
>
> Currently, the amount of memory that Giraph uses for messaging is huge.  This 
> JIRA will reduce the messaging memory by half and provide periodic updates of 
> memory for debugging.  Details are below:
> Refactored RandomMessageBenchmark to an internal vertex class.  Added 
> aggregators to RandomMessagesBenchmark to track bytes, messages, and time for 
> the messaging.  Adjusted the postSuperstep() to be called after the flush() 
> for more accurate timings.
> Added periodic minute updates for message flushing (which can take a while, 
> especially on the memory benchmark).  This helps to see how progress is going 
> and gives an ETA.
> Memory optimizations include:
> - Clear the message list after computation 
> - Free vertex messages on the source as the flush is going on 
> - TreeMap -> HashMap for VertexMutations
> - Sizing the ArrayList properly in transientInMessages

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Reply via email to