Avery Ching commented on GIRAPH-45:

Ah, I get it.  You always dump the whole map of sorted vertices and messages 
lists to a file with a BTree index and BloomFilter.  Then on read, you only 
load the initial parts of each files (BTree and BlooomFilter) to see if it has 
that vertex.

So in conclusion:

Partition-based out-of-core:
+ Only one seek per partition
- Must be able to keep all messages from any partition in memory at a given 
time, or use the recursive split solution

Sorted-map out-of-core:
+ Loads the minimum number of messages at a time into memory
- In the worst case, the number of seeks per vertex could be equal to the 
number of flushes of the sorted memory map

Does this sound about right?

If I'm correct in my understanding of your idea, then does the SortedMap<I, 
Queue<M>> concurentskiplistmap need to be sorted?  Couldn't a HashMap work?  
I'm not sure what the sorting buys you.
> Improve the way to keep outgoing messages
> -----------------------------------------
>                 Key: GIRAPH-45
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-45
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Hyunsik Choi
>            Assignee: Hyunsik Choi
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a 
> potential problem to cause out of memory when the rate of message generation 
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or 
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply via email to