Jake Mannix commented on GIRAPH-45:
So I've wanted to do this kind of thing (semi-stream the outgoing messages,
instead of buffering them all up) for practical purposes already as well. But
I've struggled with how to get it to work properly: if the messages are so big
(or so numerous) that they don't fit easily in each mapper (together with the
rest of the graph that's there), then you can help this by sending them as you
create them, but then the receiver has to buffer them on the other end...
unless the reciever is able to process them as they receive them.
But now this is a much more complex computation model: to send out messages,
you typically need to know what you've recieved in the previous step. But in
the previous step, you've been recieving messages and not *buffering them*, but
instead *processing them* in some way. If this can be implemented in the
Combiner, then great! If you can't, then what option is there except for hold
them in memory somewhere, or spill to disk somewhere?
> Improve the way to keep outgoing messages
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira