Hyunsik Choi commented on GIRAPH-12:

(a note for sharing)

Graph mutation functions (e.g., addVertexRequest, addEdgeRequest..) directly 
invoke RPC functions. 
This approach incurs RPC round-trip overheads during processing. Especially 
when many workers try to mutate vertices or edges, synchronization overheads 
may also occur in receiving sides. It may be severe as the size of cluster 

If we change graph mutation API to asynchronous messages, it would be more 
efficient. If possible, graph mutation messages and value messages (i.e., 
sendMsg) can be integrated into one message passing API.

> Investigate communication improvements
> --------------------------------------
>                 Key: GIRAPH-12
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-12
>             Project: Giraph
>          Issue Type: Improvement
>          Components: bsp
>            Reporter: Avery Ching
>            Assignee: Hyunsik Choi
>            Priority: Minor
>         Attachments: GIRAPH-12_1.patch
> Currently every worker will start up a thread to communicate with every other 
> workers.  Hadoop RPC is used for communication.  For instance if there are 
> 400 workers, each worker will create 400 threads.  This ends up using a lot 
> of memory, even with the option  
> -Dmapred.child.java.opts="-Xss64k".  
> It would be good to investigate using frameworks like Netty or custom roll 
> our own to improve this situation.  By moving away from Hadoop RPC, we would 
> also make compatibility of different Hadoop versions easier.

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


Reply via email to