[
https://issues.apache.org/jira/browse/HAMA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581352#comment-13581352
]
Thomas Jungblut edited comment on HAMA-704 at 2/19/13 3:21 PM:
---------------------------------------------------------------
If the incoming messages are already sorted by the vertex id from every peer,
it is really easy to merge them together in linear runtime and on disk. Hadoop
has quite good merging code in that regard, so I don't know if there is some
effort in reusing the code.
However, this won't fix the complete problem as the second half of the memory
is occupied by the graph itself. Maybe HAMA-732 solves it efficiently.
Oh and BTW, for fault tolerance the changing part of the graph must be
persisted after every superstep anyways. So it is just a short-term decision
not to save the graph onto a secondary storage.
was (Author: thomas.jungblut):
If the incoming messages are already sorted by the vertex id from every
peer, it is really easy to merge them together in linear runtime and on disk.
Hadoop has quite good merging code in that regard, so I don't know if there is
some effort in reusing the code.
However, this won't fix the complete problem as the second half of the memory
is occupied by the graph itself. Maybe HAMA-732 solves it efficiently.
> Optimization of memory usage during message processing
> ------------------------------------------------------
>
> Key: HAMA-704
> URL: https://issues.apache.org/jira/browse/HAMA-704
> Project: Hama
> Issue Type: Improvement
> Components: graph
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Priority: Critical
> Fix For: 0.6.1
>
> Attachments: HAMA-704.patch-v1, hama-704_v05.patch, localdisk.patch,
> mytest.patch, patch.txt, patch.txt, removeMsgMap.patch
>
>
> <vertex, message> map seems consume a lot of memory. We should figure out an
> efficient way to reduce memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
