[
https://issues.apache.org/jira/browse/HAMA-704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13579077#comment-13579077
]
Edward J. Yoon edited comment on HAMA-704 at 2/15/13 10:33 AM:
---------------------------------------------------------------
My question is whether this is really helpful. According to my simple Java
test, it looks like helpful. 1) program throws OOM exception but 2) is ok.
{code}
// 1. current TRUNK
Map<Integer, List<Integer>> map;
for(int i = 0; i < 1000000000; i++) {
msg = new ArrayList<Integer>();
for(int j = 0; j < 100000; j++) {
msg.add(j);
}
map.put(i, msg);
}
{code}
vs.
{code}
// 1. call compute() within loop
for(int i = 0; i < 1000000000; i++) {
msg = new ArrayList<Integer>();
for(int j = 0; j < 100000; j++) {
msg.add(j);
}
// i.compute(msg);
}
{code}
And, I couldn't test with huge graph yet but, performance will be decreased by
cost of sort.
With my single machine, Pagerank on 1,600,000 edges graph:
TRUNK 12 secs
TRUNK + SortedMessageQueue 39 secs
--
My patch and + SortedMessageQueue 39 secs.
was (Author: udanax):
My question is that this is really helpful. I couldn't test with huge graph
yet.
According to my simple Java test, it looks like helpful. 1) program throws OOM
exception but 2) is ok.
{code}
// 1. current TRUNK
Map<Integer, List<Integer>> map;
for(int i = 0; i < 1000000000; i++) {
msg = new ArrayList<Integer>();
for(int j = 0; j < 100000; j++) {
msg.add(j);
}
map.put(i, msg);
}
{code}
vs.
{code}
// 1. call compute() within loop
for(int i = 0; i < 1000000000; i++) {
msg = new ArrayList<Integer>();
for(int j = 0; j < 100000; j++) {
msg.add(j);
}
// i.compute(msg);
}
{code}
> Optimization of memory usage during message processing
> ------------------------------------------------------
>
> Key: HAMA-704
> URL: https://issues.apache.org/jira/browse/HAMA-704
> Project: Hama
> Issue Type: Improvement
> Components: graph
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Priority: Critical
> Fix For: 0.6.1
>
> Attachments: hama-704_v05.patch, localdisk.patch, mytest.patch,
> patch.txt, patch.txt, removeMsgMap.patch
>
>
> <vertex, message> map seems consume a lot of memory. We should figure out an
> efficient way to reduce memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
