Jakob Homan commented on GIRAPH-45:
A pull model, where the destination asks for messages as it needs them, or a
incremental push model (the first messages), seems like a bigger change than
what I was thinking of.
bq. it can make sense for the sender to hit the disk and then send them to the
other end at the beginning of next superstep
Is this predicated on the pull/incremental-push model? My thoughts weren't
based on that, but on the current shuffle-like approach. (I'd love to have a
virtual whiteboard right now.)
In a model where the outgoing messages are sent immediately, there exists the
chance that a sent message will tip the destination into having to start
spilling to disk. But that message no longer is on the originator, so it
relieves pressure on the originator and lessens the chance the originator will
need to do so. So, one worker spills and the message hits disk once.
If, on the other hand, we do not send outgoing messages immediately and instead
spill on the originator, when the message finally is sent, it also has the
potential to tip the destination into spilling. If we're lucky, the
destination doesn't spill and only one worker spills and the message hits disk
once. If we're unlucky and the destination has to spill, then we have two
workers spilling and the message hitting disk twice.
> Improve the way to keep outgoing messages
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
For more information on JIRA, see: http://www.atlassian.com/software/jira