[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170859#comment-13170859 ] Hudson commented on GIRAPH-57: -- Integrated in Giraph-trunk-Commit #49 (See [https://builds.apache.org/job/Giraph-trunk-Commit/49/]) Ancillary to GIRAPH-57 (type parameters of I cannot be determined). aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1215070 Files : * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff, GIRAPH-57.diff.2 > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170857#comment-13170857 ] Avery Ching commented on GIRAPH-57: --- Emergency fix to allow trunk to compile on certain platforms: [ERROR] /home/hudson/hudson-slave/workspace/Giraph-trunk-Commit/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java:[66,45] type parameters of I cannot be determined; no unique maximal instance exists for type variable I with upper bounds I,org.apache.hadoop.io.WritableComparable == --- incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java (original) +++ incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java Fri Dec 16 09:26:44 2011 @@ -63,7 +63,7 @@ public class VertexIdMessagescreateVertexIndex(getConf()); > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff, GIRAPH-57.diff.2 > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170602#comment-13170602 ] Avery Ching commented on GIRAPH-57: --- I only ran mvn package =(. I need to remember to do mvn install next time. Sorry. > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170605#comment-13170605 ] Jakob Homan commented on GIRAPH-57: --- Can we post the final patch, along with the "I give this to Apache" button? > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170601#comment-13170601 ] Claudio Martella commented on GIRAPH-57: I guess how it passed mvn install... > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170586#comment-13170586 ] Hudson commented on GIRAPH-57: -- Integrated in Giraph-trunk-Commit #48 (See [https://builds.apache.org/job/Giraph-trunk-Commit/48/]) GIRAPH-57: Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together. (aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1214983 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/CODE_CONVENTIONS * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170532#comment-13170532 ] jirapos...@reviews.apache.org commented on GIRAPH-57: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3201/#review3928 --- Ship it! Looks cool and straight from here. All those CODE_CONVENTIONS fixes made it looks scary from far. - Claudio On 2011-12-14 19:50:29, Avery Ching wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/3201/ bq. --- bq. bq. (Updated 2011-12-14 19:50:29) bq. bq. bq. Review request for giraph. bq. bq. bq. Summary bq. --- bq. bq. Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements). bq. bq. Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS. bq. bq. bq. This addresses bug GIRAPH-57. bq. https://issues.apache.org/jira/browse/GIRAPH-57 bq. bq. bq. Diffs bq. - bq. bq.http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java PRE-CREATION bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java PRE-CREATION bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849 bq. bq. Diff: https://reviews.apache.org/r/3201/diff bq. bq. bq. Testing bq. --- bq. bq. Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster. bq. bq. bq. Thanks, bq. bq. Avery bq. bq. > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13170489#comment-13170489 ] Avery Ching commented on GIRAPH-57: --- Anyone wanna review? This is a pretty straightforward change with obvious improvements in performance... > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169630#comment-13169630 ] jirapos...@reviews.apache.org commented on GIRAPH-57: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3201/ --- (Updated 2011-12-14 19:50:29.358317) Review request for giraph. Changes --- Forgot a few important files (VertexIdMessages.java and VertexIdMessagesList.java) Summary --- Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements). Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS. This addresses bug GIRAPH-57. https://issues.apache.org/jira/browse/GIRAPH-57 Diffs (updated) - http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java PRE-CREATION http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849 Diff: https://reviews.apache.org/r/3201/diff Testing --- Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster. Thanks, Avery > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > Attachments: GIRAPH-57.diff > > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169627#comment-13169627 ] jirapos...@reviews.apache.org commented on GIRAPH-57: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3201/ --- Review request for giraph. Summary --- Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs during a flush. This improves the number of RPC / sec and overall I/O bandwidth. The amount of batching is done by the total number of messages and is configurable at runtime (default of 5000, weighted toward helping smaller messages). I have noted some performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent improvements). Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces between 'synchronized' and '('. Removed spaces and standardized in the CODE_CONVENTIONS. This addresses bug GIRAPH-57. https://issues.apache.org/jira/browse/GIRAPH-57 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1214406 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java 1213849 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1213849 Diff: https://reviews.apache.org/r/3201/diff Testing --- Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small cluster. Thanks, Avery > Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together > > > Key: GIRAPH-57 > URL: https://issues.apache.org/jira/browse/GIRAPH-57 > Project: Giraph > Issue Type: Improvement >Reporter: Jakob Homan >Assignee: Avery Ching > > Right now messages are sent to a vertex one at a time. It would be good to > have a putMsgs call that could send messages to multiple vertices (all hosted > on the same worker). We'd save a huge number of individual RPC calls at the > expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira