Jakob Homan commented on GIRAPH-45:
Yeah, this would involve changing the "save all the messages up and send them
at the end of the superstep" model to a more streaming model for outgoing
messages, which I think may provide more throughput overall. If we send
messages as they are generated (or in bunches intermittently), we'll decrease
the time spent "shuffling" them at the end of step. This is important as the
"shuffle" is a peer-to-peer process with every worker potentially sending
something to every other worker and, in a chatty computation, may put pressure
on the network.
I'm skeptical how often combiners will be both possible and actually
implemented. If a combiner is not defined, is there any benefit at all to not
immediately sending out the non-local messages? One approach is when memory
starts getting tight (or on a regular schedule), run the combiner and then ship
out non-local-bound messages to their workers. Similarly, the receiving worker
can run its combiner on the messages as they come in, spilling when necessary.
It will probably be quite expensive to un-spill to disk to re-run the combiner,
assuming the combiner managed to have an effect.
> Improve the way to keep outgoing messages
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira