Re: some giraph code
I have already been in email contact with Eugenia, because a colleague of mine is currently at UCI. I'll cc this mail to her :) --sebastian On 06.01.2012 14:20, Claudio Martella wrote: Hello guys, today I stumbled upon this: https://grape.ics.uci.edu/wiki/asterix/raw-attachment/wiki/cs295-2011-fall-ProjectTeams/Team_8_Report.pdf the code can be found here: http://code.google.com/p/graph-algorithm-ports-giraph-hyracks/ Was anybody aware of this? Do you think we could take advantage of this code?
[jira] [Updated] (GIRAPH-118) Clarify messages behavior in BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-118: Description: initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. Affects Version/s: 0.70.0 Assignee: Claudio Martella Clarify messages behavior in BasicVertex Key: GIRAPH-118 URL: https://issues.apache.org/jira/browse/GIRAPH-118 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Priority: Minor initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-118) Clarify messages behavior in BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181340#comment-13181340 ] Claudio Martella commented on GIRAPH-118: - As a user with some experience with Giraph, it took a while to rationalize the process while working on GIRAPH-45, I guess a new user should be a bit confused as well. Clarify messages behavior in BasicVertex Key: GIRAPH-118 URL: https://issues.apache.org/jira/browse/GIRAPH-118 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Priority: Minor initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: some giraph code
Hi Sebastian, Great. It might be good to either (a) copy giraph-dev@ on the discussion to keep the rest of the community outside of the specific sub-group you're talking to aware of whats up; or (b) send a summary of the discussion here. It's important to demonstrate that discussion within Apache Giraph is happening *on the mailing lists* here. Cheers, Chris On Jan 6, 2012, at 8:25 AM, Sebastian Schelter wrote: I have already been in email contact with Eugenia, because a colleague of mine is currently at UCI. I'll cc this mail to her :) --sebastian On 06.01.2012 14:20, Claudio Martella wrote: Hello guys, today I stumbled upon this: https://grape.ics.uci.edu/wiki/asterix/raw-attachment/wiki/cs295-2011-fall-ProjectTeams/Team_8_Report.pdf the code can be found here: http://code.google.com/p/graph-algorithm-ports-giraph-hyracks/ Was anybody aware of this? Do you think we could take advantage of this code? ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
Re: some giraph code
@Chris I have to clarify this. I haven't been discussing Giraph-specific issues outside of the mailinglist, I just know Eugenia and know that she is working on porting some algorithms onto Pregel-like platforms. @Eugenia Sorry for the confusion :) We (the Giraph community) stumbled upon your current work at https://grape.ics.uci.edu/wiki/asterix/raw-attachment/wiki/cs295-2011-fall-ProjectTeams/Team_8_Report.pdf. It would be great if you could share your experiences and tell us about the problems you encountered (especially the design decisions you're criticizing). If you want to, it would also be great if you could contribute back your code. Best, Sebastian On 06.01.2012 15:22, Mattmann, Chris A (388J) wrote: Hi Sebastian, Great. It might be good to either (a) copy giraph-dev@ on the discussion to keep the rest of the community outside of the specific sub-group you're talking to aware of whats up; or (b) send a summary of the discussion here. It's important to demonstrate that discussion within Apache Giraph is happening *on the mailing lists* here. Cheers, Chris On Jan 6, 2012, at 8:25 AM, Sebastian Schelter wrote: I have already been in email contact with Eugenia, because a colleague of mine is currently at UCI. I'll cc this mail to her :) --sebastians On 06.01.2012 14:20, Claudio Martella wrote: Hello guys, today I stumbled upon this: https://grape.ics.uci.edu/wiki/asterix/raw-attachment/wiki/cs295-2011-fall-ProjectTeams/Team_8_Report.pdf the code can be found here: http://code.google.com/p/graph-algorithm-ports-giraph-hyracks/ Was anybody aware of this? Do you think we could take advantage of this code? ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++
some weird code
Hello,
I hope somebody can shed some light on a piece of code i'm looking at
while working on GIRAPH-45 (and this code is also the object of
GIRAPH-95, so we'd probably get two birds with one stone here).
The code is taking care of vertex resolving in
BasicRPCCommunication::prepareSuperstep():
[line 1091]:
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
First, vertex cannot be null as it's resolved by vertexRevolver, but i
guess it's a sanity check. But the real question is: why would you
setVertex() considering it's been already initialized correctly in
vertexResolver?
Am I missing something or did I just realize that GIRAPH-95 is solved
by just removing that line? :)
Thanks
--
Claudio Martella
claudio.marte...@gmail.com
[jira] [Created] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
[ https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-119: Attachment: GIRAPH-119.diff Trivial refactor to solve the issue. VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-119.diff Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
[ https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181522#comment-13181522 ] Jakob Homan commented on GIRAPH-119: +1. The change log isn't usually modified as part of the patch, but as part of the commit, although I don't see a reason it would hurt, except perhaps conflicts in the file? VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-119.diff Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
Ok, I didn't know. I usually do it as it requires 0-knowledge by a second hand. Btw, why would it conflict? On Fri, Jan 6, 2012 at 8:05 PM, Jakob Homan (Commented) (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181522#comment-13181522 ] Jakob Homan commented on GIRAPH-119: +1. The change log isn't usually modified as part of the patch, but as part of the commit, although I don't see a reason it would hurt, except perhaps conflicts in the file? VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-119.diff Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira -- Claudio Martella claudio.marte...@gmail.com
Re: [jira] [Commented] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
Maybe if changelog got modified (unlikely, I know). Doesn't hurt anything. Also, for non-committer contributors, they shouldn't modify the log since they don't know who may commit their code (ie bob via joe) On Fri, Jan 6, 2012 at 11:07 AM, Claudio Martella claudio.marte...@gmail.com wrote: Ok, I didn't know. I usually do it as it requires 0-knowledge by a second hand. Btw, why would it conflict? On Fri, Jan 6, 2012 at 8:05 PM, Jakob Homan (Commented) (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181522#comment-13181522 ] Jakob Homan commented on GIRAPH-119: +1. The change log isn't usually modified as part of the patch, but as part of the commit, although I don't see a reason it would hurt, except perhaps conflicts in the file? VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-119.diff Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira -- Claudio Martella claudio.marte...@gmail.com
[jira] [Commented] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
[ https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181541#comment-13181541 ] Hudson commented on GIRAPH-119: --- Integrated in Giraph-trunk-Commit #61 (See [https://builds.apache.org/job/Giraph-trunk-Commit/61/]) GIRAPH-119: VertexCombiner should work on IterableM instead of ListM claudio : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228356 Files : * /incubator/giraph/trunk/CHANGELOG * /incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java * /incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/VertexCombiner.java * /incubator/giraph/trunk/src/test/java/org/apache/giraph/TestVertexTypes.java VertexCombiner should work on IterableM instead of ListM Key: GIRAPH-119 URL: https://issues.apache.org/jira/browse/GIRAPH-119 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-119.diff Currently VertexCombiner expects a ListM. It should be refactored to IterableM to sync with Iterable-based BasicVertex messages logics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: some weird code
Hi Claudio, answers inline:
On 1/6/12 8:25 AM, Claudio Martella wrote:
Hello,
I hope somebody can shed some light on a piece of code i'm looking at
while working on GIRAPH-45 (and this code is also the object of
GIRAPH-95, so we'd probably get two birds with one stone here).
The code is taking care of vertex resolving in
BasicRPCCommunication::prepareSuperstep():
[line 1091]:
if (vertex != null) {
((MutableVertexI, V, E, M) vertex).setVertexId(vertexIndex);
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else if (originalVertex != null) {
partition.removeVertex(originalVertex.getVertexId());
}
First, vertex cannot be null as it's resolved by vertexRevolver, but i
guess it's a sanity check. But the real question is: why would you
setVertex() considering it's been already initialized correctly in
vertexResolver?
Actually it can be null. Since user's can implement their own vertex
resolver, they are allowed to return null from the javadoc.
/**
* A vertex may have been removed, created zero or more times and had
* zero or more messages sent to it. This method will handle all
situations
* excluding the normal case (a vertex already exists and has zero
or more
* messages sent it to).
*
* @param vertexId Vertex id (can be used for {@link BasicVertex}'s
*initialize())
* @param vertex Original vertex or null if none
* @param vertexChanges Changes that happened to this vertex or
null if none
* @param messages messages received in the last superstep or null
if none
* @return Vertex to be returned, if null, and a vertex currently
exists
* it will be removed
*/
Am I missing something or did I just realize that GIRAPH-95 is solved
by just removing that line? :)
Thanks
Well, not sure about that. The set is done there I think to ensure
safety. Here's the issue: Suppose that the resolve() doesn't set the
vertex id correctly (i.e. in this partition). That would be a bug and
probably cause issues. Probably this should be changed to be a check
though. Something like...
if (vertex != null) {
if (vertex.getVertexId().equals(vertexIndex)) {
throw new
IllegalStateException(BasicRPCCommunications: Illegal to set the vertex
index differently from + vertexIndex);
if (originalVertex == null) {
partition.putVertex((BasicVertexI, V, E, M) vertex);
} else {
partition.removeVertex(originalVertex.getVertexId());
}
}
What do you think?
Avery
[jira] [Commented] (GIRAPH-119) VertexCombiner should work on IterableM instead of ListM
[
https://issues.apache.org/jira/browse/GIRAPH-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181566#comment-13181566
]
Avery Ching commented on GIRAPH-119:
Minor nit
@Override
public FloatWritable combine(LongWritable vertexIndex,
- ListFloatWritable msgList)
+ IterableFloatWritable msgList)
throws IOException {
return null;
}
@@ -97,7 +97,7 @@ public class TestVertexTypes
@Override
public DoubleWritable combine(LongWritable vertexIndex,
- ListDoubleWritable msgList)
+ IterableDoubleWritable msgList)
throws IOException {
return null;
}
probably should have changed msgList to messages or something like that. Not a
big deal. =)
VertexCombiner should work on IterableM instead of ListM
Key: GIRAPH-119
URL: https://issues.apache.org/jira/browse/GIRAPH-119
Project: Giraph
Issue Type: Improvement
Components: graph
Affects Versions: 0.70.0
Reporter: Claudio Martella
Assignee: Claudio Martella
Attachments: GIRAPH-119.diff
Currently VertexCombiner expects a ListM. It should be refactored to
IterableM to sync with Iterable-based BasicVertex messages logics.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-118) Clarify messages behavior in BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181570#comment-13181570 ] Avery Ching commented on GIRAPH-118: Seems reasonable. Please make sure to update the javadoc and the MutableVertex implementations. Clarify messages behavior in BasicVertex Key: GIRAPH-118 URL: https://issues.apache.org/jira/browse/GIRAPH-118 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Priority: Minor initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-118) Clarify messages behavior in BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-118: Attachment: GIRAPH-119.diff Apparently the initialize() issue is also true for other parameters as well such as the edges (and outside of the documentation also vertexId: I'm looking at TestEdgeListVertex i.e.). With this little one I just touched the putMessages() issue, probably we can think about the initialize() later. Clarify messages behavior in BasicVertex Key: GIRAPH-118 URL: https://issues.apache.org/jira/browse/GIRAPH-118 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Priority: Minor Attachments: GIRAPH-119.diff initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-118) Clarify messages behavior in BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-118: Attachment: GIRAPH-118.diff Messed up with issue number in patch filename, sorry :) Clarify messages behavior in BasicVertex Key: GIRAPH-118 URL: https://issues.apache.org/jira/browse/GIRAPH-118 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Priority: Minor Attachments: GIRAPH-118.diff, GIRAPH-119.diff initialize() can receive a null parameter for messages (at least that's what EdgeListVertex does). We should avoid that and pass an empty Iterable instead. That should be cheap for us inside of the InputFormat, just passing a static immutable empty list. setMessages(IterableM) should be changed to putMessages(IterableM). the set prefix suggests an assignment, while setMessages is used to transfer the messages to the internal datastructure the user is responsible for. putMessages() should clarify this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
