Review Request: GIRAPH-34 Failure of Vertex reflection for putVertexList from GIRAPH-27
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1932/ --- Review request for giraph. Summary --- The problem shows up when messages are sent locally, and therefore not reflected. Hence, after the message has been delivered to the other local vertex, in the next superstep, if the vertex who sent the message modifies the object, it will modify the message before the receiving vertex has a chance to access it. Also, combiners maybe operate on the (shared) message simultaneously with the compute() method, therefore potential issues could also occur with messages eventually sent to remote workers. The solution I am proposing is that all messages should be copied to prevent this error from occuring as it is pretty tough to debug. An alternative would just be to tell users to create copies themselves, but it's less convenient and not intuitive in my opinion. This addresses bug GIRAPH-34. https://issues.apache.org/jira /browse/GIRAPH-34 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java 1171389 Diff: https://reviews.apache.org/r/1932/diff Testing --- I passed local unittests (mvn package) and unittests on my local machine's Hadop instance (mvn package -Dprop.mapred.job.tracker=localhost:50300). Thanks, Avery
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106684#comment-13106684 ] Avery Ching commented on GIRAPH-34: --- I've figured it out! Can someone please review https://reviews.apache.org/r/1932/? Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106685#comment-13106685 ] jirapos...@reviews.apache.org commented on GIRAPH-34: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1932/ --- Review request for giraph. Summary --- The problem shows up when messages are sent locally, and therefore not reflected. Hence, after the message has been delivered to the other local vertex, in the next superstep, if the vertex who sent the message modifies the object, it will modify the message before the receiving vertex has a chance to access it. Also, combiners maybe operate on the (shared) message simultaneously with the compute() method, therefore potential issues could also occur with messages eventually sent to remote workers. The solution I am proposing is that all messages should be copied to prevent this error from occuring as it is pretty tough to debug. An alternative would just be to tell users to create copies themselves, but it's less convenient and not intuitive in my opinion. This addresses bug GIRAPH-34. https://issues.apache.org/jira /browse/GIRAPH-34 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java 1171389 Diff: https://reviews.apache.org/r/1932/diff Testing --- I passed local unittests (mvn package) and unittests on my local machine's Hadop instance (mvn package -Dprop.mapred.job.tracker=localhost:50300). Thanks, Avery Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106717#comment-13106717 ] Jake Mannix commented on GIRAPH-34: --- Wait, why would the sending Vertex modify the message object they just sent? Why would the even have a reference to it anymore? It's a message, right? Could we not simply document that messages should be treated as ephemeral and not retained? It seems like doing a bunch of reflection and object copying for each message to be sent could get prohibitively expensive. As I look through the VertexRangeBalance code, I notice also that VertexList extends ArrayListWritableVertexI, V, E, M. Yikes! Not everything needs to be a Vertex anymore - if we let people extend BasicVertex (or MutableVertex) instead of always extending Vertex, they'll get killed with runtime classcast exceptions if they try to do any balancing. Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106739#comment-13106739 ] Avery Ching commented on GIRAPH-34: --- Suppose for example, they send their vertex value to the destination vertex. I.e. sendMsg(destVertex, getVertexValue()), then since the vertex value is not copied and delivered, the message will have all the changes of the vertex value after it's supposedly been sent (not good). I think it will be hard to force users to ensure that the sent messages must be an object they are sure to not modify. That's similar to how this error happened in the first place (See SimpleCheckpointVertex). I really don't see how we can avoid copying the object as it will be pretty dangerous when mutable types are used as messages. The changes from GIRAPH-27 didn't cause that issue, it's a bug that has likely been around for awhile. Glad to have it reported by Christian. As you mentioned, there could be issues elsewhere with the change to not require everything to subclass Vertex. Maybe address in a later JIRA? Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106748#comment-13106748 ] Jake Mannix commented on GIRAPH-34: --- I'll definitely open another JIRA for the Vertex subclasses, and dig into that a bit. But on this current topic, I see how users could possibly do something like sendMsg(destVertex, getVertexValue()), yes. But isn't this analogous to in regular Hadoop-land, that you simply cannot expect to hang onto your Writable instances and use them later. If you're in Mapper.map(SomethingWritableComparable key, SomethingWritable value, Context c), you should *never* just buffer up the key and value instances, as this is practically guaranteed to break - Hadoop will be re-using the key and value as container objects to read new bytes off of disk for the next invocation to map(), so that java objects are rarely created, instead you're just constantly doing simple bit/byte operations on the disk stream, and setting values inside of Writable containers. It seems like one of the basic contracts of Writables (at least in Hadoop-land) is that they are always to be considered containers: call get() or getSomeKindOfThing() on them as soon as you have a handle on one, and use whatever *that* is, assuming that the framework can and will reuse your original Writable. Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106769#comment-13106769 ] Dmitriy V. Ryaboy commented on GIRAPH-34: - Would it make sense to make the messages immutable in sendMsg? Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-34 Failure of Vertex reflection for putVertexList from GIRAPH-27
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1932/ --- (Updated 2011-09-16 20:09:21.003725) Review request for giraph. Changes --- Instead of copying the user messages, simply rely on Javadoc to inform users of the contract with sent message values. Passed the failing unittest with local Hadoop instance and local unittests. Summary --- The problem shows up when messages are sent locally, and therefore not reflected. Hence, after the message has been delivered to the other local vertex, in the next superstep, if the vertex who sent the message modifies the object, it will modify the message before the receiving vertex has a chance to access it. Also, combiners maybe operate on the (shared) message simultaneously with the compute() method, therefore potential issues could also occur with messages eventually sent to remote workers. The solution I am proposing is that all messages should be copied to prevent this error from occuring as it is pretty tough to debug. An alternative would just be to tell users to create copies themselves, but it's less convenient and not intuitive in my opinion. This addresses bug GIRAPH-34. https://issues.apache.org/jira /browse/GIRAPH-34 Diffs (updated) - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java 1171389 Diff: https://reviews.apache.org/r/1932/diff Testing --- I passed local unittests (mvn package) and unittests on my local machine's Hadop instance (mvn package -Dprop.mapred.job.tracker=localhost:50300). Thanks, Avery
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106771#comment-13106771 ] jirapos...@reviews.apache.org commented on GIRAPH-34: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1932/ --- (Updated 2011-09-16 20:09:21.003725) Review request for giraph. Changes --- Instead of copying the user messages, simply rely on Javadoc to inform users of the contract with sent message values. Passed the failing unittest with local Hadoop instance and local unittests. Summary --- The problem shows up when messages are sent locally, and therefore not reflected. Hence, after the message has been delivered to the other local vertex, in the next superstep, if the vertex who sent the message modifies the object, it will modify the message before the receiving vertex has a chance to access it. Also, combiners maybe operate on the (shared) message simultaneously with the compute() method, therefore potential issues could also occur with messages eventually sent to remote workers. The solution I am proposing is that all messages should be copied to prevent this error from occuring as it is pretty tough to debug. An alternative would just be to tell users to create copies themselves, but it's less convenient and not intuitive in my opinion. This addresses bug GIRAPH-34. https://issues.apache.org/jira /browse/GIRAPH-34 Diffs (updated) - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java 1171389 http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java 1171389 Diff: https://reviews.apache.org/r/1932/diff Testing --- I passed local unittests (mvn package) and unittests on my local machine's Hadop instance (mvn package -Dprop.mapred.job.tracker=localhost:50300). Thanks, Avery Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106779#comment-13106779 ] Avery Ching commented on GIRAPH-34: --- Making the message immutable, is that more than javadoc? It seems pretty difficult and possibly expensive to guarantee that an object is immutable http://stackoverflow.com/questions/203475/how-do-i-identify-immutable-objects-in-java Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106780#comment-13106780 ] Jake Mannix commented on GIRAPH-34: --- Yeah, how do you do that, Dmitriy? Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106789#comment-13106789 ] Dmitriy V. Ryaboy commented on GIRAPH-34: - Never mind, Writable is an envelope for the actual message, and it doesn't matter what we do to writable -- the concern here is calling methods on the contained message, and of course we can't control that. +1 Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106798#comment-13106798 ] Avery Ching commented on GIRAPH-34: --- Jake your thoughts on the current diff? My next flight leaves in a few hours =)... Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106804#comment-13106804 ] Jake Mannix commented on GIRAPH-34: --- +1 from me - although I haven't run it on an actual cluster, so I'm going by my reading of the code. Although we should think further about ways we can be safe: it's possible that the right and efficient thing to do is analogous to your context.write() example: we take the Writable message, and we serialize the Writable to a byte[], and pass that byte[] to the local recipient if there is one. That recipient should be able to inexpensively deserialize and rehydrate the messages on the fly when running the VertexCombiner (only using one container Writable at a time, doing the same thing that Hadoop does, essentially) and just before the call to compute(). Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avery Ching resolved GIRAPH-34. --- Resolution: Fixed I've committed with +1's from Dmitriy and Jake. We can think about a better way to be safe in a future JIRA. Although Jake, how much more expensive it is to serialize/deserialize to/from a byte[] vs instantiating the object and using Hadoop's ReflectionUtils.copy()? Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-27) Mutable static global state in Vertex.java should be refactored
[ https://issues.apache.org/jira/browse/GIRAPH-27?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106815#comment-13106815 ] Hudson commented on GIRAPH-27: -- Integrated in Giraph-trunk-Commit #7 (See [https://builds.apache.org/job/Giraph-trunk-Commit/7/]) GIRAPH-34: Failure of Vertex reflection for putVertexList from GIRAPH-27. (aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171776 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java * /incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java Mutable static global state in Vertex.java should be refactored --- Key: GIRAPH-27 URL: https://issues.apache.org/jira/browse/GIRAPH-27 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Jake Mannix Assignee: Jake Mannix Attachments: GIRAPH-27.patch, GIRAPH-27.patch Vertex.java has a bunch of static methods for getting/setting global graph state (total number of vertices, edges, a reference to the GraphMapper, etc). Refactoring this into a GraphState object, which every Vertex can hold onto a reference to (yes, a tiny bit more memory per Vertex, but in comparison to what's already in there...) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-34) Failure of Vertex reflection for putVertexList from GIRAPH-27
[ https://issues.apache.org/jira/browse/GIRAPH-34?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106816#comment-13106816 ] Hudson commented on GIRAPH-34: -- Integrated in Giraph-trunk-Commit #7 (See [https://builds.apache.org/job/Giraph-trunk-Commit/7/]) GIRAPH-34: Failure of Vertex reflection for putVertexList from GIRAPH-27. (aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1171776 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/Vertex.java * /incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexRangeBalancer.java Failure of Vertex reflection for putVertexList from GIRAPH-27 -- Key: GIRAPH-34 URL: https://issues.apache.org/jira/browse/GIRAPH-34 Project: Giraph Issue Type: Bug Reporter: Christian Kunz Assignee: Avery Ching Attachments: GIRAPH-34.patch Christian actually found this bug. I am filing the JIRA on his behalf. Here's my error when running TestVertexRangeBalancer. java.lang.RuntimeException: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:768) at org.apache.giraph.graph.BspServiceWorker.exchangeVertexRanges(BspServiceWorker.java:1282) at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:589) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) Caused by: java.io.IOException: Call to returnwhose-lm/10.72.107.231:30002 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) at $Proxy3.putVertexList(Unknown Source) at org.apache.giraph.comm.BasicRPCCommunications.sendVertexListReq(BasicRPCCommunications.java:766) ... 10 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.ipc.Client$Connection.receiveResponse(Client.java:767) at org.apache.hadoop.ipc.Client$Connection.run(Client.java:712) I identified and fixed the issue by making BasicVertex implement Configurable and making the graph state set in BasicRPCCommunications. There is one more error though that I'll try and solve before putting up a reviewboard. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-11) Improve the graph distribution of Giraph
[ https://issues.apache.org/jira/browse/GIRAPH-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106839#comment-13106839 ] Avery Ching commented on GIRAPH-11: --- Hi Severin, as far as getting started, feel free to take a look at any of the open issues (https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=truejqlQuery=project+%3D+GIRAPH+AND+status+%3D+Open+ORDER+BY+priority+DESCmode=hide) that are unassigned. Also, if you think of any interesting features to add, file a JIRA. We'd be happy to have you contribute =). Improve the graph distribution of Giraph Key: GIRAPH-11 URL: https://issues.apache.org/jira/browse/GIRAPH-11 Project: Giraph Issue Type: Improvement Reporter: Avery Ching Assignee: Avery Ching Currently, Giraph assumes that the data from the VertexInputFormat is sorted. If the user data is not sorted by the vertex id, they must first run a MapReduce or Pig job to generate a sorted dataset. This is often a bit inconvenient. Giraph graph partitioning is currently range based and there are some advantages and disadvantages of this approach. The proposal of this JIRA would be to allow for both range and hash based partitioning and provide more flexibility to the user. Design goals for the graph distribution: * Allow vertices to be unordered or unordered * Ability to repartition * Select the partitioning scheme based on user needs (i.e. hash or range based) * Ability to provide user-specific hints about partitions Hash-based partitioning * Good vertex balancing across ranges for random data * Bad at vertex id locality Range-based partitioning * Good at vertex id locality * Ability to split ranges easily * Can cause hotspots for hot ranges -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-12) Investigate communication improvements
[ https://issues.apache.org/jira/browse/GIRAPH-12?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13106842#comment-13106842 ] Jake Mannix commented on GIRAPH-12: --- Hey Hyunsik, if you're going to write a benchmark for the RPC stuff, that would be totally great. I'd like to start playing around with trying Finagle in here, and we can compare notes on what kinds of techniques among both approaches work better, unless I'd be stepping on your toes by doing so... Investigate communication improvements -- Key: GIRAPH-12 URL: https://issues.apache.org/jira/browse/GIRAPH-12 Project: Giraph Issue Type: Improvement Components: bsp Reporter: Avery Ching Assignee: Hyunsik Choi Priority: Minor Attachments: GIRAPH-12_1.patch Currently every worker will start up a thread to communicate with every other workers. Hadoop RPC is used for communication. For instance if there are 400 workers, each worker will create 400 threads. This ends up using a lot of memory, even with the option -Dmapred.child.java.opts=-Xss64k. It would be good to investigate using frameworks like Netty or custom roll our own to improve this situation. By moving away from Hadoop RPC, we would also make compatibility of different Hadoop versions easier. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-36) Ensure that subclassing BasicVertex is possible by user apps
Ensure that subclassing BasicVertex is possible by user apps Key: GIRAPH-36 URL: https://issues.apache.org/jira/browse/GIRAPH-36 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Jake Mannix Assignee: Jake Mannix Priority: Blocker Fix For: 0.70.0 Original assumptions in Giraph were that all users would subclass Vertex (which extended MutableVertex extended BasicVertex). Classes which wish to have application specific data structures (ie. not a TreeMapI, EdgeI,E) may need to extend either MutableVertex or BasicVertex. Unfortunately VertexRange extends ArrayListVertex, and there are other places where the assumption is that vertex classes are either Vertex, or at least MutableVertex. Let's make sure the internal APIs allow for BasicVertex to be the base class. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
