[
https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171273#comment-13171273
]
Avery Ching commented on GIRAPH-45:
-----------------------------------
Ah, I get it. You always dump the whole map of sorted vertices and messages
lists to a file with a BTree index and BloomFilter. Then on read, you only
load the initial parts of each files (BTree and BlooomFilter) to see if it has
that vertex.
So in conclusion:
Partition-based out-of-core:
+ Only one seek per partition
- Must be able to keep all messages from any partition in memory at a given
time, or use the recursive split solution
Sorted-map out-of-core:
+ Loads the minimum number of messages at a time into memory
- In the worst case, the number of seeks per vertex could be equal to the
number of flushes of the sorted memory map
Does this sound about right?
If I'm correct in my understanding of your idea, then does the SortedMap<I,
Queue<M>> concurentskiplistmap need to be sorted? Couldn't a HashMap work?
I'm not sure what the sorting buys you.
> Improve the way to keep outgoing messages
> -----------------------------------------
>
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
