[
https://issues.apache.org/jira/browse/GIRAPH-262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418781#comment-13418781
]
Eli Reisman commented on GIRAPH-262:
------------------------------------
nice, I was thinking of the local message change myself for a JIRA!
After playing with GIRAPH-256 a bunch and trying to use that to regulate the
size of outgoing collections of vertices after being read from input splits, I
am seeing that allowing too big a message to go through or too many small ones
both overwhelm the netty buffers. Would it be useful to find a way to weigh
outgoing objects of every sort and simply tune the netty buffers to handle
those sizes? That, and get them out of serialization before they pile up?
Perhaps this (before the calculations but during message passing or vertex
passing at any super step) is where spill to disk stuff could be handy? I am
having a lot of trouble getting the settings right and consistent to get
through different data loads with my memory constraints.
I'll try this patch for sure and see if it helps, the rotator might be just the
thing to stop message pile ups. I keep seeing netty get overwhelmed on the
Decode side.
> Netty optimizations and bandwidth tracking
> ------------------------------------------
>
> Key: GIRAPH-262
> URL: https://issues.apache.org/jira/browse/GIRAPH-262
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-262.patch
>
>
> * Makes netty the default instead of HadoopRPC
> * Added optimization to handle requests locally when possible rather than go
> over the network
> * Added TimedLogger to print only within a given time period
> * Added optimization for using multiple channels between clients/servers when
> bandwidth is limited per connection
> * Added ByteCounter to track bandwidth across Netty (can be later integrated
> with GIRAPH-232).
> * Upgraded rat to 0.8 and excluded iml files (Intellij Idea)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
