[jira] [Created] (GIRAPH-109) GiraphRunner should provide support for combiners
GiraphRunner should provide support for combiners - Key: GIRAPH-109 URL: https://issues.apache.org/jira/browse/GIRAPH-109 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Currently there's no way to tell GiraphRunner that you want to use a Combiner. A simple option should be added, similar to the way in- and outputformats are specified. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-110) Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance
[ https://issues.apache.org/jira/browse/GIRAPH-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173470#comment-13173470 ] Hudson commented on GIRAPH-110: --- Integrated in Giraph-trunk-Commit #55 (See [https://builds.apache.org/job/Giraph-trunk-Commit/55/]) GIRAPH-110: Add guide to setup the enviroment for running the unittests in a pseudo-distributed hadoop instance. (ssc via aching) aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1221461 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/README Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance Key: GIRAPH-110 URL: https://issues.apache.org/jira/browse/GIRAPH-110 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Assignee: Sebastian Schelter Priority: Minor Fix For: 0.70.0 Attachments: GIRAPH-110.2.patch, GIRAPH-110.patch Giraph should provide a small guide for setting up the local environment to run the unit tests in a pseudo-distributed hadoop instance as there are some non-obvious hurdles to take. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-108) Refactor code to run independently of Map/Reduce
[
https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173497#comment-13173497
]
Ed Kohlwey commented on GIRAPH-108:
---
{quote}
is probably not going to fly. I'd rather not introduce anything more hacky than
is strictly necessary. I'll try to take a look at this code by the end of the
week.
{quote}
I agree. My philosophy in this patch is to balance incremental change against
dropping a patch bomb. I thought about tearing up the Input and Output format
classes but realized that it would probably make the patch gigantic (and thus
less likely to be accepted).
I went ahead and started a discussion on GIRAPH-111 for this.
Refactor code to run independently of Map/Reduce
Key: GIRAPH-108
URL: https://issues.apache.org/jira/browse/GIRAPH-108
Project: Giraph
Issue Type: Improvement
Components: graph
Reporter: Ed Kohlwey
Attachments: GIRAPH-108
It would be nice for Giraph to be refactored such that the code could
eventually be run outside of map/reduce. This will allow people to write
drivers that can run in the cool new resource manager frameworks like Mesos
and YARN, and eventually let the application's code base evolve to be
independent of map/reduce.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173512#comment-13173512 ] Avery Ching commented on GIRAPH-111: I'm not clear on why this is necessary. Couldn't we simply call the I/O methods as Hadoop would when we're not using Hadoop? Am I missing something? Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-111) Refactor I/O to be independent of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173537#comment-13173537 ] Jakob Homan commented on GIRAPH-111: bq. I'm not clear on why this is necessary. I agree. Hadoop's file formats, etc. are designed to be exceedingly forgiving and flexible as to the underlying storage mechanism. Can you point to where they're lacking for Mesos' case? bq. We could also copy out the relevant Hadoop I/O classes (InputFormat, OutputFormat, etc) into Giraph, rename their packages, and begin reworking them in an appropriate way to better suit Giraph. -1. Therein lies madness. Refactor I/O to be independent of Map/Reduce Key: GIRAPH-111 URL: https://issues.apache.org/jira/browse/GIRAPH-111 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey The I/O mechanisms should probably be abstracted entirely from Map/Reduce in order to support making Giraph an independent framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-110) Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance
[ https://issues.apache.org/jira/browse/GIRAPH-110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173631#comment-13173631 ] Jakob Homan commented on GIRAPH-110: Sorry to be late on this one, but I'd been meaning to ask if we should retire most of the README content in favor of the site documentation? The content between the two was originally duplicated and is starting to drift... Add guide to setup the enviroment for running the unit tests in a pseudo-distributed hadoop instance Key: GIRAPH-110 URL: https://issues.apache.org/jira/browse/GIRAPH-110 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Assignee: Sebastian Schelter Priority: Minor Fix For: 0.70.0 Attachments: GIRAPH-110.2.patch, GIRAPH-110.patch Giraph should provide a small guide for setting up the local environment to run the unit tests in a pseudo-distributed hadoop instance as there are some non-obvious hurdles to take. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
Change cast to Vertex used in prepareSuperstep() to BasicVertex
---
Key: GIRAPH-113
URL: https://issues.apache.org/jira/browse/GIRAPH-113
Project: Giraph
Issue Type: Bug
Reporter: Yuanyuan Tian
Priority: Minor
Hi,
I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it
uses more compact and efficient mahout collections. However I run into an error
when running the algorithm:
java.lang.ClassCastException:
org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to
org.apache.giraph.graph.Vertex
at
org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016)
at
org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728)
... 7 more
Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(),
the LongDoubleFloatDoubleVertex are cast to Vertex in the following code
fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead of
Vertex.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((VertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The
problem went away, and the algorithm finished without any error. But I am not
sure this change has any implication to other parts of the code. So, I hope to
get some comments from the Giraph developers.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[
https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Avery Ching reassigned GIRAPH-113:
--
Assignee: Avery Ching
Change cast to Vertex used in prepareSuperstep() to BasicVertex
---
Key: GIRAPH-113
URL: https://issues.apache.org/jira/browse/GIRAPH-113
Project: Giraph
Issue Type: Bug
Reporter: Yuanyuan Tian
Assignee: Avery Ching
Priority: Minor
Hi,
I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it
uses more compact and efficient mahout collections. However I run into an
error when running the algorithm:
java.lang.ClassCastException:
org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to
org.apache.giraph.graph.Vertex
at
org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016)
at
org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728)
... 7 more
Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(),
the LongDoubleFloatDoubleVertex are cast to Vertex in the following code
fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead
of Vertex.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((VertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The
problem went away, and the algorithm finished without any error. But I am not
sure this change has any implication to other parts of the code. So, I hope
to get some comments from the Giraph developers.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[
https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Avery Ching updated GIRAPH-113:
---
Attachment: GIRAPH-113.patch
Change cast to Vertex used in prepareSuperstep() to BasicVertex
---
Key: GIRAPH-113
URL: https://issues.apache.org/jira/browse/GIRAPH-113
Project: Giraph
Issue Type: Bug
Reporter: Yuanyuan Tian
Assignee: Avery Ching
Priority: Minor
Attachments: GIRAPH-113.patch
Hi,
I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it
uses more compact and efficient mahout collections. However I run into an
error when running the algorithm:
java.lang.ClassCastException:
org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to
org.apache.giraph.graph.Vertex
at
org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016)
at
org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728)
... 7 more
Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(),
the LongDoubleFloatDoubleVertex are cast to Vertex in the following code
fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead
of Vertex.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((VertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The
problem went away, and the algorithm finished without any error. But I am not
sure this change has any implication to other parts of the code. So, I hope
to get some comments from the Giraph developers.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[
https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173921#comment-13173921
]
Jakob Homan commented on GIRAPH-113:
+1 (grumbling about GIRAPH-83)
Change cast to Vertex used in prepareSuperstep() to BasicVertex
---
Key: GIRAPH-113
URL: https://issues.apache.org/jira/browse/GIRAPH-113
Project: Giraph
Issue Type: Bug
Reporter: Yuanyuan Tian
Assignee: Avery Ching
Priority: Minor
Attachments: GIRAPH-113.patch
Hi,
I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it
uses more compact and efficient mahout collections. However I run into an
error when running the algorithm:
java.lang.ClassCastException:
org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to
org.apache.giraph.graph.Vertex
at
org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016)
at
org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728)
... 7 more
Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(),
the LongDoubleFloatDoubleVertex are cast to Vertex in the following code
fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead
of Vertex.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((VertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The
problem went away, and the algorithm finished without any error. But I am not
sure this change has any implication to other parts of the code. So, I hope
to get some comments from the Giraph developers.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-113) Change cast to Vertex used in prepareSuperstep() to BasicVertex
[
https://issues.apache.org/jira/browse/GIRAPH-113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173924#comment-13173924
]
Hudson commented on GIRAPH-113:
---
Integrated in Giraph-trunk-Commit #56 (See
[https://builds.apache.org/job/Giraph-trunk-Commit/56/])
GIRAPH-113: Change cast to Vertex used in prepareSuperstep() to
BasicVertex. (humming80 via aching)
aching : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1221634
Files :
* /incubator/giraph/trunk/CHANGELOG
*
/incubator/giraph/trunk/src/main/java/org/apache/giraph/comm/BasicRPCCommunications.java
Change cast to Vertex used in prepareSuperstep() to BasicVertex
---
Key: GIRAPH-113
URL: https://issues.apache.org/jira/browse/GIRAPH-113
Project: Giraph
Issue Type: Bug
Reporter: Yuanyuan Tian
Assignee: Avery Ching
Priority: Minor
Attachments: GIRAPH-113.patch
Hi,
I decided to use LongDoubleFloatDoubleVertex in a graph algorithm because it
uses more compact and efficient mahout collections. However I run into an
error when running the algorithm:
java.lang.ClassCastException:
org.apache.giraph.graph.LongDoubleFloatDoubleVertex cannot be cast to
org.apache.giraph.graph.Vertex
at
org.apache.giraph.comm.BasicRPCCommunications.prepareSuperstep(BasicRPCCommunications.java:1016)
at
org.apache.giraph.graph.BspServiceWorker.startSuperstep(BspServiceWorker.java:843)
at org.apache.giraph.graph.GraphMapper.map(GraphMapper.java:569)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:728)
... 7 more
Basically, the problem is that in BasicRPCCommunications.prepareSuperStep(),
the LongDoubleFloatDoubleVertex are cast to Vertex in the following code
fragment. But LongDoubleFloatDoubleVertex inherits from BasicVertex instead
of Vertex.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((VertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
I did a simple change: cast LongDoubleFloatDoubleVertex to BasicVertex. The
problem went away, and the algorithm finished without any error. But I am not
sure this change has any implication to other parts of the code. So, I hope
to get some comments from the Giraph developers.
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: GIRAPH-112: Use elements() properly in LongDoubleFloatDoubleVertex
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/ --- Review request for giraph. Summary --- As pointed out by YuanYua, the array returned by elements() cannot have its length used since the array contains all the elements currently stored in the mahout collections, even including invalid elements between size and capacity. Whenever possible I converted elements() into forEach(), forEachKey(), forEachPair(). Used size() in other cases. Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java. This addresses bug GIRAPH-112. https://issues.apache.org/jira/browse/GIRAPH-112 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1221634 Diff: https://reviews.apache.org/r/3287/diff Testing --- Local unittests and MR unittests. Thanks, Avery
[jira] [Commented] (GIRAPH-108) Refactor code to run independently of Map/Reduce
[ https://issues.apache.org/jira/browse/GIRAPH-108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173931#comment-13173931 ] Avery Ching commented on GIRAPH-108: Actually, I'll let Jakob take a first crack at looking at this since he's got some expertise in the area. Refactor code to run independently of Map/Reduce Key: GIRAPH-108 URL: https://issues.apache.org/jira/browse/GIRAPH-108 Project: Giraph Issue Type: Improvement Components: graph Reporter: Ed Kohlwey Attachments: GIRAPH-108 It would be nice for Giraph to be refactored such that the code could eventually be run outside of map/reduce. This will allow people to write drivers that can run in the cool new resource manager frameworks like Mesos and YARN, and eventually let the application's code base evolve to be independent of map/reduce. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-112) Use elements() properly in LongDoubleFloatDoubleVertex
[
https://issues.apache.org/jira/browse/GIRAPH-112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13173932#comment-13173932
]
jirapos...@reviews.apache.org commented on GIRAPH-112:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3287/
---
Review request for giraph.
Summary
---
As pointed out by YuanYua, the array returned by elements() cannot have its
length used since the array contains all the elements currently stored in the
mahout collections, even including invalid elements between size and capacity.
Whenever possible I converted elements() into forEach(), forEachKey(),
forEachPair(). Used size() in other cases.
Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java.
This addresses bug GIRAPH-112.
https://issues.apache.org/jira/browse/GIRAPH-112
Diffs
-
http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java
1221634
Diff: https://reviews.apache.org/r/3287/diff
Testing
---
Local unittests and MR unittests.
Thanks,
Avery
Use elements() properly in LongDoubleFloatDoubleVertex
--
Key: GIRAPH-112
URL: https://issues.apache.org/jira/browse/GIRAPH-112
Project: Giraph
Issue Type: Bug
Components: graph
Affects Versions: 0.70.0
Environment: Any
Reporter: Yuanyuan Tian
Fix For: 0.70.0
Attachments: GIRAPH-112.patch
Original Estimate: 5m
Remaining Estimate: 5m
I found a bug in LongDoubleFloatDoubleVertex.write(DataOutput out) when
running a small graph algorithm. The symptom is that a vertex read from a
different worker becomes junk after the RPC communication. And the source of
the problem is the writing of the messages in
LongDoubleFloatDoubleVertex.write(DataOutput out):
for(double msg : messageList.elements()) {
out.writeDouble(msg);
}
Here messageList.elements() will returns all the elements currently stored in
the mahout DoubleArrayList, even including invalid elements between size and
capacity. Therefore, the write() function will write a bunch of invalid
messages, which will cause error when reading them back in readfields().
The following is a simple solution:
double[] elements=messageList.elements();
for(int i=0; imessageList.size(); i++) {
out.writeDouble(elements[i]);
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-112: Use elements() properly in LongDoubleFloatDoubleVertex
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/#review4036 --- Ship it! I ran into the same issue yesterday and the solution presented here is correct. For reasons of efficiency, list.elements() returns the internal underlying array for the list, which might be bigger than the number of elements stored in the list. Therefore you should only iterate until list.size() or use the foreachKey() callback. - Sebastian On 2011-12-21 07:50:20, Avery Ching wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/3287/ --- (Updated 2011-12-21 07:50:20) Review request for giraph. Summary --- As pointed out by YuanYua, the array returned by elements() cannot have its length used since the array contains all the elements currently stored in the mahout collections, even including invalid elements between size and capacity. Whenever possible I converted elements() into forEach(), forEachKey(), forEachPair(). Used size() in other cases. Fixed some formatting violations as well in LongDoubleFloatDoubleVertex.java. This addresses bug GIRAPH-112. https://issues.apache.org/jira/browse/GIRAPH-112 Diffs - http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java 1221634 Diff: https://reviews.apache.org/r/3287/diff Testing --- Local unittests and MR unittests. Thanks, Avery
