[
https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157588#comment-13157588
]
Claudio Martella commented on GIRAPH-45:
----------------------------------------
was thinking, what about storing the out-of-core messages with leveldb?
writing them as they come should be quite efficient as it writes data to an
append-only commit log.
it keeps data sorted, so reading with an iterator should be very fast.
each worker could have for each superstep a leveldb database and each message
could have a key composed like this: <destination vertex id>:<uuid>, keep in
the payload the message.
the db could be destroyed and created at each superstep by each worker.
as far as combiners are concerned, they could be called (one the second
endpoint) after each get() from the db at compute() time. We're trying to save
memory space, not disk. They would be processed before each vertex's compute()
and you'd be required to have all the messages sent to that vertex (in that
superstep) in memory, because combiners' can't work incrementally.
what do you guys think?
> Improve the way to keep outgoing messages
> -----------------------------------------
>
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
