[jira] [Commented] (GIRAPH-11) Improve the graph distribution of Giraph
[ https://issues.apache.org/jira/browse/GIRAPH-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148187#comment-13148187 ] Hyunsik Choi commented on GIRAPH-11: That's a huge patch :) I have just started to explore your patch. I will leave some comments (maybe tomorrow). > Improve the graph distribution of Giraph > > > Key: GIRAPH-11 > URL: https://issues.apache.org/jira/browse/GIRAPH-11 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.70.0 >Reporter: Avery Ching >Assignee: Avery Ching > Attachments: GIRAPH-11.diff > > > Currently, Giraph assumes that the data from the VertexInputFormat is sorted. > If the user data is not sorted by the vertex id, they must first run a > MapReduce or Pig job to generate a sorted dataset. This is often a bit > inconvenient. > Giraph graph partitioning is currently range based and there are some > advantages and disadvantages of this approach. The proposal of this JIRA > would be to allow for both range and hash based partitioning and provide more > flexibility to the user. > Design goals for the graph distribution: > * Allow vertices to be unordered or unordered > * Ability to repartition > * Select the partitioning scheme based on user needs (i.e. hash or range > based) > * Ability to provide user-specific hints about partitions > Hash-based partitioning > * Good vertex balancing across ranges for random data > * Bad at vertex id locality > Range-based partitioning > * Good at vertex id locality > * Ability to split ranges easily > * Can cause hotspots for hot ranges -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148080#comment-13148080 ] Claudio Martella commented on GIRAPH-74: I've always noted an ambiguity in the API. There's the halt boolean field which is there and is set by voteToHalt(), but then there's the isHalted() which can potentially lead to a conflictual implementation of the concept (the user might not know about the private field halt). So I propose: we either put the setHalted() to BasicVertex and give an implementation to Vertex so the user can ignore it, or we get rid of both isHalted and setHalted together. I don't see why we should also get rid of voteToHalt(), it has a strong semantical meaning and matches the original Pregel API. I agree though that the return value of compute might do that as well. So: I vote for getting rid of isHalted and setHalted. > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147975#comment-13147975 ] Avery Ching commented on GIRAPH-74: --- Well, if you vote to halt, your compute() is never run again (unless you get a message and your state is reset at a later superstep). The halted state after compute() is called is the one that is final for the vertex for that particular superstep. Another way of doing this could be to have compute return a boolean and get rid of setHalted() and voteToHalt(). > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147972#comment-13147972 ] Jake Mannix commented on GIRAPH-74: --- But my concern is regarding how this works with general BSP algorithm correctness: if you can vote to halt, then later cancel it, can this lead to problems, or is this generally allowed? > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147968#comment-13147968 ] Avery Ching commented on GIRAPH-74: --- Either the vertex will keep track of it or the user will (have their own boolean and then decide whether to vote to halt or not). I don't have a strong opinion. > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147948#comment-13147948 ] Jake Mannix commented on GIRAPH-74: --- In general, I'd prefer setHalted(boolean), but this allows a Vertex to effectively cancel its vote to halt at a later time. Should that be allowed? > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147919#comment-13147919 ] Avery Ching commented on GIRAPH-74: --- Some of this I also addressed in GIRAPH-11 =). I think combining the getter/setter behavior is a little strange. Perhaps void setHalted(boolean isHalted); boolean isHalted(); Anyone else have an opinion? > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-11) Improve the graph distribution of Giraph
[ https://issues.apache.org/jira/browse/GIRAPH-11?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147916#comment-13147916 ] Avery Ching commented on GIRAPH-11: --- Sooo...anyone care to look at this? I'd really like to get it since the old graph partitioning sucks. One we have it in and test a bit, hopefully we can cut a release now.. > Improve the graph distribution of Giraph > > > Key: GIRAPH-11 > URL: https://issues.apache.org/jira/browse/GIRAPH-11 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.70.0 >Reporter: Avery Ching >Assignee: Avery Ching > Attachments: GIRAPH-11.diff > > > Currently, Giraph assumes that the data from the VertexInputFormat is sorted. > If the user data is not sorted by the vertex id, they must first run a > MapReduce or Pig job to generate a sorted dataset. This is often a bit > inconvenient. > Giraph graph partitioning is currently range based and there are some > advantages and disadvantages of this approach. The proposal of this JIRA > would be to allow for both range and hash based partitioning and provide more > flexibility to the user. > Design goals for the graph distribution: > * Allow vertices to be unordered or unordered > * Ability to repartition > * Select the partitioning scheme based on user needs (i.e. hash or range > based) > * Ability to provide user-specific hints about partitions > Hash-based partitioning > * Good vertex balancing across ranges for random data > * Bad at vertex id locality > Range-based partitioning > * Good at vertex id locality > * Ability to split ranges easily > * Can cause hotspots for hot ranges -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-73) A little refactoring
[ https://issues.apache.org/jira/browse/GIRAPH-73?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147913#comment-13147913 ] Avery Ching commented on GIRAPH-73: --- Hi Sebastian, Thanks for the refactoring and for taking a look at the code. Have you looked at the CODE_CONVENTIONS in the base directory, I see a little bit of weird indenting (not a big deal). After a quick pass, looks like you have make some nice improvements in the code. Now that we have added Guava, why not take advantage as you have suggested? Also, regarding - DoubleWritable> vertex = BspUtils.createVertex(getContext().getConfiguration()); + DoubleWritable> vertex = BspUtils.createVertex(getContext().getConfiguration()); We have had issues when types aren't specified. See http://stackoverflow.com/questions/2640060/inconsistency-between-sun-jre-javac-and-eclipse-java-compiler and an issue (https://issues.apache.org/jira/browse/GIRAPH-27). Also, were you able to pass unittests locally (mvn test) and on a actual Hadoop instance (mvn test -Dprop.mapred.job.tracker=:)? > A little refactoring > > > Key: GIRAPH-73 > URL: https://issues.apache.org/jira/browse/GIRAPH-73 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.70.0 >Reporter: Sebastian Schelter >Priority: Minor > Attachments: GIRAPH-73.patch > > > Hi, I'm currently reading Giraph's sources and starting to play with it. I > fixed some small things along the way (like making sure writers are closed, > exceptions are logged, etc.), thought that maybe helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Newbie wanting to get involved with Giraph development
Claudio gave great advice. The only thing I'd like to add is that you also might want to consider using reviewboard for throwing up patches (https://reviews.apache.org/) as it helps to get line-by-line comments and suggestions. Definitely email the mailing lists if you have questions and have fun! Avery On 11/10/11 9:40 AM, Claudio Martella wrote: Hi, I don't know if there's anything such as an official Development Process, I can share how I usually do when I contribute to ASF projects. (1) Considering there's already an open issue (a ticket in the JIRA) I'd download from SVN the version to which the issue applies, I'd write the fix, test it through unittests that come with the project (and fix them if necessary) and create the patch (svn diff> ISSUE.diff is the way I do) that I later attach it to the issue. (2) If there's not open issue yet about a bug or a feature missing, if it's trivial (such as the one you've chosen) I'd write the patch as in (1) and open the issue describing the problem by attaching the fix. If it's not trivial I'd open the issue and wait for some contribution to the discussion. Learn from the other issues you find on the JIRA. Make sure to follow CODE_CONVENTIONS file ( ;-) ). I'm expecting the experts to give more insights about the process. Hope this helps, Claudio On Thu, Nov 10, 2011 at 5:56 PM, Shaunak Kashyap wrote: Hi, I'm a newbie to Giraph and ASF projects in general. I would like to help with Giraph development and think I've found the perfect JIRA to start: https://issues.apache.org/jira/browse/GIRAPH-63. Before I make any code changes, however, I'd like to know more about the development process of this project. What is a good place to start learning about this? Thank you, Shaunak -- "Now the hardness of this world slowly grinds your dreams away / Makin' a fool's joke out of the promises we make" --- Bruce Springsteen, "Blood Brothers"
Re: Newbie wanting to get involved with Giraph development
Hi, I don't know if there's anything such as an official Development Process, I can share how I usually do when I contribute to ASF projects. (1) Considering there's already an open issue (a ticket in the JIRA) I'd download from SVN the version to which the issue applies, I'd write the fix, test it through unittests that come with the project (and fix them if necessary) and create the patch (svn diff > ISSUE.diff is the way I do) that I later attach it to the issue. (2) If there's not open issue yet about a bug or a feature missing, if it's trivial (such as the one you've chosen) I'd write the patch as in (1) and open the issue describing the problem by attaching the fix. If it's not trivial I'd open the issue and wait for some contribution to the discussion. Learn from the other issues you find on the JIRA. Make sure to follow CODE_CONVENTIONS file ( ;-) ). I'm expecting the experts to give more insights about the process. Hope this helps, Claudio On Thu, Nov 10, 2011 at 5:56 PM, Shaunak Kashyap wrote: > Hi, > > I'm a newbie to Giraph and ASF projects in general. I would like to > help with Giraph development and think I've found the perfect JIRA to > start: https://issues.apache.org/jira/browse/GIRAPH-63. > > Before I make any code changes, however, I'd like to know more about > the development process of this project. What is a good place to start > learning about this? > > Thank you, > > Shaunak > > -- > "Now the hardness of this world slowly grinds your dreams away / > Makin' a fool's joke out of the promises we make" --- Bruce > Springsteen, "Blood Brothers" > -- Claudio Martella claudio.marte...@gmail.com
Newbie wanting to get involved with Giraph development
Hi, I'm a newbie to Giraph and ASF projects in general. I would like to help with Giraph development and think I've found the perfect JIRA to start: https://issues.apache.org/jira/browse/GIRAPH-63. Before I make any code changes, however, I'd like to know more about the development process of this project. What is a good place to start learning about this? Thank you, Shaunak -- "Now the hardness of this world slowly grinds your dreams away / Makin' a fool's joke out of the promises we make" --- Bruce Springsteen, "Blood Brothers"
[jira] [Created] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex --- Key: GIRAPH-74 URL: https://issues.apache.org/jira/browse/GIRAPH-74 Project: Giraph Issue Type: Bug Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Fix For: 0.70.0 Attachments: GIRAPH-36-cleaup.diff After GIRAPH-36, in a couple of places Vertex was left instead of BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean isHalted(boolean state) method that totally hides boolean halt field. I believe this unifies the voteToHalt() and isHalted() methods totally hiding the halting management of Vertices (before it was relying on the existence of boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-74) GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex
[ https://issues.apache.org/jira/browse/GIRAPH-74?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Claudio Martella updated GIRAPH-74: --- Attachment: GIRAPH-36-cleaup.diff This fixes the issue as in the issue description. > GIRAPH-36 missed a couple of liners when passing from Vertex to BasicVertex > --- > > Key: GIRAPH-74 > URL: https://issues.apache.org/jira/browse/GIRAPH-74 > Project: Giraph > Issue Type: Bug > Components: graph >Affects Versions: 0.70.0 >Reporter: Claudio Martella >Assignee: Claudio Martella > Fix For: 0.70.0 > > Attachments: GIRAPH-36-cleaup.diff > > > After GIRAPH-36, in a couple of places Vertex was left instead of > BasicVertex. To achieve the fix a BasicVertex was extended with a new boolean > isHalted(boolean state) method that totally hides boolean halt field. I > believe this unifies the voteToHalt() and isHalted() methods totally hiding > the halting management of Vertices (before it was relying on the existence of > boolean halt field). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-73) A little refactoring
[ https://issues.apache.org/jira/browse/GIRAPH-73?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Schelter updated GIRAPH-73: - Attachment: GIRAPH-73.patch > A little refactoring > > > Key: GIRAPH-73 > URL: https://issues.apache.org/jira/browse/GIRAPH-73 > Project: Giraph > Issue Type: Improvement >Affects Versions: 0.70.0 >Reporter: Sebastian Schelter >Priority: Minor > Attachments: GIRAPH-73.patch > > > Hi, I'm currently reading Giraph's sources and starting to play with it. I > fixed some small things along the way (like making sure writers are closed, > exceptions are logged, etc.), thought that maybe helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-73) A little refactoring
A little refactoring Key: GIRAPH-73 URL: https://issues.apache.org/jira/browse/GIRAPH-73 Project: Giraph Issue Type: Improvement Affects Versions: 0.70.0 Reporter: Sebastian Schelter Priority: Minor Hi, I'm currently reading Giraph's sources and starting to play with it. I fixed some small things along the way (like making sure writers are closed, exceptions are logged, etc.), thought that maybe helpful. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
