Jakob Homan commented on GIRAPH-45:

I'm suggesting that messages should only be spilled to disk on their target 
local disk.  Spilling to disk would still be the main way to relieve memory 
pressure (along with combiners, of course).  The question is should you be 
responsible to spilling messages that you won't be processing in the next 
superstep? If so, we're spending more time writing and reading the messages on 
the originator (and then, possibly, writing and reading again on the 
destination worker).  
bq. But now this is a much more complex computation model: to send out 
messages, you typically need to know what you've recieved in the previous step.
I'm afraid I don't follow.  This is still consistent with the BSP model. In a 
compute iteration, I can send a message to Z and it can either be held until 
everyone else is done computing, or immediately sent to the worker responsible 
for Z.  Either way it gets stored in a container until the next superstep 
starts.  No actual processing is done on the message itself.
> Improve the way to keep outgoing messages
> -----------------------------------------
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a 
> potential problem to cause out of memory when the rate of message generation 
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or 
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply via email to