[
https://issues.apache.org/jira/browse/HAMA-258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13018865#comment-13018865
]
Thomas Jungblut edited comment on HAMA-258 at 4/12/11 3:58 PM:
---------------------------------------------------------------
Looks good, but we should provide readers and writers as described in the
Pregel paper.
These logics are just messing up the code of a BSP class.
Output:
We can provide a output collector on each peer.
I'm currently writing a blog post for partitioning and splitting, -I'll post it
then.-
http://codingwiththomas.blogspot.com/2011/04/apache-hama-partitioning.html
Basically it provides just a split for each groom with the modulo function :
vertexId % sizeOfCluster.
Then it just get's writtin into a SequenceFile and the groom can access it.
This is going to be a more scalable solution, what if the messages are too
large for the ram of the groom. Most people just have a small cluster but many
data. We should take this into account.
was (Author: thomas.jungblut):
Looks good, but we should provide readers and writers as described in the
Pregel paper.
These logics are just messing up the code of a BSP class.
Output:
We can provide a output collector on each peer.
I'm currently writing a blog post for partitioning and splitting, I'll post it
then.
Basically it provides just a split for each groom with the modulo function :
vertexId % sizeOfCluster.
Then it just get's writtin into a SequenceFile and the groom can access it.
This is going to be a more scalable solution, what if the messages are too
large for the ram of the groom. Most people just have a small cluster but many
data. We should take this into account.
> Design a input and output system
> --------------------------------
>
> Key: HAMA-258
> URL: https://issues.apache.org/jira/browse/HAMA-258
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: Edward J. Yoon
> Fix For: 0.3.0
>
>
> This issue will handle the input and output system with data splitter.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
