[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[
https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170857#comment-13170857
]
Avery Ching commented on GIRAPH-57:
---
Emergency fix to allow trunk to compile on certain platforms:
[ERROR]
/home/hudson/hudson-slave/workspace/Giraph-trunk-Commit/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java:[66,45]
type parameters of II cannot be determined; no unique maximal instance
exists for type variable I with upper bounds
I,org.apache.hadoop.io.WritableComparable
==
---
incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java
(original)
+++
incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java
Fri Dec 16 09:26:44 2011
@@ -63,7 +63,7 @@ public class VertexIdMessagesI extends
@Override
public void readFields(DataInput input) throws IOException {
-vertexId = BspUtils.createVertexIndex(getConf());
+vertexId = BspUtils.IcreateVertexIndex(getConf());
Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
Key: GIRAPH-57
URL: https://issues.apache.org/jira/browse/GIRAPH-57
Project: Giraph
Issue Type: Improvement
Reporter: Jakob Homan
Assignee: Avery Ching
Attachments: GIRAPH-57.diff, GIRAPH-57.diff.2
Right now messages are sent to a vertex one at a time. It would be good to
have a putMsgs call that could send messages to multiple vertices (all hosted
on the same worker). We'd save a huge number of individual RPC calls at the
expense of having smaller calls with larger payloads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[
https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170532#comment-13170532
]
jirapos...@reviews.apache.org commented on GIRAPH-57:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/#review3928
---
Ship it!
Looks cool and straight from here. All those CODE_CONVENTIONS fixes made it
looks scary from far.
- Claudio
On 2011-12-14 19:50:29, Avery Ching wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/3201/
bq. ---
bq.
bq. (Updated 2011-12-14 19:50:29)
bq.
bq.
bq. Review request for giraph.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Added new putVertexIdMessagesList RPC and supporting classes
(VertexIdMessages and VertexIdMessagesList) to reduce the total number of RPCs
during a flush. This improves the number of RPC / sec and overall I/O
bandwidth. The amount of batching is done by the total number of messages and
is configurable at runtime (default of 5000, weighted toward helping smaller
messages). I have noted some performance results in
https://issues.apache.org/jira/browse/GIRAPH-57 (between 25 - 1075 percent
improvements).
bq.
bq. Also, while tinkering with BasicRPCCommunications, notices inconsistent
spaces between 'synchronized' and '('. Removed spaces and standardized in the
CODE_CONVENTIONS.
bq.
bq.
bq. This addresses bug GIRAPH-57.
bq. https://issues.apache.org/jira/browse/GIRAPH-57
bq.
bq.
bq. Diffs
bq. -
bq.
bq.http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS
1213849
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java
1213849
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1214406
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
1213849
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java
PRE-CREATION
bq.
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1213849
bq.
bq. Diff: https://reviews.apache.org/r/3201/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a
small cluster.
bq.
bq.
bq. Thanks,
bq.
bq. Avery
bq.
bq.
Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
Key: GIRAPH-57
URL: https://issues.apache.org/jira/browse/GIRAPH-57
Project: Giraph
Issue Type: Improvement
Reporter: Jakob Homan
Assignee: Avery Ching
Attachments: GIRAPH-57.diff
Right now messages are sent to a vertex one at a time. It would be good to
have a putMsgs call that could send messages to multiple vertices (all hosted
on the same worker). We'd save a huge number of individual RPC calls at the
expense of having smaller calls with larger payloads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170586#comment-13170586 ] Hudson commented on GIRAPH-57: -- Integrated in Giraph-trunk-Commit #48 (See [https://builds.apache.org/job/Giraph-trunk-Commit/48/]) GIRAPH-57: Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together. (aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1214983 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/CODE_CONVENTIONS * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessages.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/VertexIdMessagesList.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together Key: GIRAPH-57 URL: https://issues.apache.org/jira/browse/GIRAPH-57 Project: Giraph Issue Type: Improvement Reporter: Jakob Homan Assignee: Avery Ching Attachments: GIRAPH-57.diff Right now messages are sent to a vertex one at a time. It would be good to have a putMsgs call that could send messages to multiple vertices (all hosted on the same worker). We'd save a huge number of individual RPC calls at the expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170601#comment-13170601 ] Claudio Martella commented on GIRAPH-57: I guess how it passed mvn install... Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together Key: GIRAPH-57 URL: https://issues.apache.org/jira/browse/GIRAPH-57 Project: Giraph Issue Type: Improvement Reporter: Jakob Homan Assignee: Avery Ching Attachments: GIRAPH-57.diff Right now messages are sent to a vertex one at a time. It would be good to have a putMsgs call that could send messages to multiple vertices (all hosted on the same worker). We'd save a huge number of individual RPC calls at the expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[ https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13170605#comment-13170605 ] Jakob Homan commented on GIRAPH-57: --- Can we post the final patch, along with the I give this to Apache button? Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together Key: GIRAPH-57 URL: https://issues.apache.org/jira/browse/GIRAPH-57 Project: Giraph Issue Type: Improvement Reporter: Jakob Homan Assignee: Avery Ching Attachments: GIRAPH-57.diff Right now messages are sent to a vertex one at a time. It would be good to have a putMsgs call that could send messages to multiple vertices (all hosted on the same worker). We'd save a huge number of individual RPC calls at the expense of having smaller calls with larger payloads. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-57) Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
[
https://issues.apache.org/jira/browse/GIRAPH-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169627#comment-13169627
]
jirapos...@reviews.apache.org commented on GIRAPH-57:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3201/
---
Review request for giraph.
Summary
---
Added new putVertexIdMessagesList RPC and supporting classes (VertexIdMessages
and VertexIdMessagesList) to reduce the total number of RPCs during a flush.
This improves the number of RPC / sec and overall I/O bandwidth. The amount of
batching is done by the total number of messages and is configurable at runtime
(default of 5000, weighted toward helping smaller messages). I have noted some
performance results in https://issues.apache.org/jira/browse/GIRAPH-57 (between
25 - 1075 percent improvements).
Also, while tinkering with BasicRPCCommunications, notices inconsistent spaces
between 'synchronized' and '('. Removed spaces and standardized in the
CODE_CONVENTIONS.
This addresses bug GIRAPH-57.
https://issues.apache.org/jira/browse/GIRAPH-57
Diffs
-
http://svn.apache.org/repos/asf/incubator/giraph/trunk/CODE_CONVENTIONS
1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/ArrayListWritable.java
1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
1214406
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/CommunicationsInterface.java
1213849
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java
1213849
Diff: https://reviews.apache.org/r/3201/diff
Testing
---
Passed local and Hadoop unittests. Used the RandomMessageBenchmark on a small
cluster.
Thanks,
Avery
Add new RPC call (putVertexIdMessagesList) to batch putMsgList RPCs together
Key: GIRAPH-57
URL: https://issues.apache.org/jira/browse/GIRAPH-57
Project: Giraph
Issue Type: Improvement
Reporter: Jakob Homan
Assignee: Avery Ching
Right now messages are sent to a vertex one at a time. It would be good to
have a putMsgs call that could send messages to multiple vertices (all hosted
on the same worker). We'd save a huge number of individual RPC calls at the
expense of having smaller calls with larger payloads.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
