[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13477674#comment-13477674 ] Thomas Jungblut commented on HAMA-599: -- Since message queues are copied various times and require huge disk accesses we are going to make it more scalable via HAMA-644. > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458542#comment-13458542 ] Edward J. Yoon commented on HAMA-599: - Since memory issue of graph job will be fixed by Thomas's HAMA-642, let's remove my dirty multi-step partitioning code in graph module if there's no problem w/ Hadoop RPC. > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444726#comment-13444726 ] Edward J. Yoon commented on HAMA-599: - Test passed. I just commited my changes. > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[
https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443951#comment-13443951
]
Edward J. Yoon commented on HAMA-599:
-
{quote}I actually hoped to address this with the diskqueue.{quote}
Me too but I'm not sure what is the best way. I'll test and share results.
> Improvement of network-based runtime partitioner
>
>
> Key: HAMA-599
> URL: https://issues.apache.org/jira/browse/HAMA-599
> Project: Hama
> Issue Type: Improvement
> Components: bsp core, graph
>Affects Versions: 0.5.0
>Reporter: Edward J. Yoon
>Assignee: Edward J. Yoon
> Fix For: 0.6.0
>
> Attachments: multi-stepPartitioning.patch
>
>
> Current runtime partitioning in graph is really inefficient and melodramatic.
> This issue improves code or figures out another way. See also HAMA-531
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[
https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443901#comment-13443901
]
Edward J. Yoon commented on HAMA-599:
-
{code}
attempt_201208290116_0002_04_0: 12/08/29 01:18:16 ERROR bsp.BSPTask:
Shutting down ping service.
attempt_201208290116_0002_04_0: 12/08/29 01:18:16 FATAL bsp.GroomServer:
Error running child
attempt_201208290116_0002_04_0: java.util.ConcurrentModificationException
attempt_201208290116_0002_04_0: at
java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
attempt_201208290116_0002_04_0: at
java.util.HashMap$ValueIterator.next(HashMap.java:822)
attempt_201208290116_0002_04_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:487)
attempt_201208290116_0002_04_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:172)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1271)
attempt_201208290116_0002_04_0: java.util.ConcurrentModificationException
attempt_201208290116_0002_04_0: at
java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
attempt_201208290116_0002_04_0: at
java.util.HashMap$ValueIterator.next(HashMap.java:822)
attempt_201208290116_0002_04_0: at
org.apache.hama.graph.GraphJobRunner.loadVertices(GraphJobRunner.java:487)
attempt_201208290116_0002_04_0: at
org.apache.hama.graph.GraphJobRunner.setup(GraphJobRunner.java:172)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.BSPTask.runBSP(BSPTask.java:165)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.BSPTask.run(BSPTask.java:143)
attempt_201208290116_0002_04_0: at
org.apache.hama.bsp.GroomServer$BSPPeerChild.main(GroomServer.java:1271)
attempt_201208290116_0002_04_0: log4j:WARN No appenders could be found for
logger (org.apache.avro.ipc.NettyTransceiver).
attempt_201208290116_0002_04_0: log4j:WARN Please initialize the log4j
system properly.
attempt_201208290116_0002_04_0: log4j:WARN See
http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
{code}
> Improvement of network-based runtime partitioner
>
>
> Key: HAMA-599
> URL: https://issues.apache.org/jira/browse/HAMA-599
> Project: Hama
> Issue Type: Improvement
> Components: bsp core, graph
>Affects Versions: 0.5.0
>Reporter: Edward J. Yoon
>Assignee: Edward J. Yoon
> Fix For: 0.6.0
>
> Attachments: multi-stepPartitioning.patch
>
>
> Current runtime partitioning in graph is really inefficient and melodramatic.
> This issue improves code or figures out another way. See also HAMA-531
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443873#comment-13443873 ] Thomas Jungblut commented on HAMA-599: -- Yeah, basically repairing can be paired with partitioning. So it automatically completes the graph. The underlying problem is that our messaging is not scalable, I actually hoped to address this with the diskqueue. > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443862#comment-13443862 ] Edward J. Yoon commented on HAMA-599: - Just noticed that repairing also problematic. P.S., it's only a temporary patch, not the fundamental solution for the problem. As Leonidas mentioned too[1], 1. http://markmail.org/thread/qtd22bpdydakmkk7 > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-599) Improvement of network-based runtime partitioner
[ https://issues.apache.org/jira/browse/HAMA-599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13443818#comment-13443818 ] Thomas Jungblut commented on HAMA-599: -- Looks good to me, +1. > Improvement of network-based runtime partitioner > > > Key: HAMA-599 > URL: https://issues.apache.org/jira/browse/HAMA-599 > Project: Hama > Issue Type: Improvement > Components: bsp core, graph >Affects Versions: 0.5.0 >Reporter: Edward J. Yoon >Assignee: Edward J. Yoon > Fix For: 0.6.0 > > Attachments: multi-stepPartitioning.patch > > > Current runtime partitioning in graph is really inefficient and melodramatic. > This issue improves code or figures out another way. See also HAMA-531 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
