[jira] [Commented] (GIRAPH-47) Export Worker's Context/State to vertices through pre/post/Application/Superstep
[ https://issues.apache.org/jira/browse/GIRAPH-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145694#comment-13145694 ] Avery Ching commented on GIRAPH-47: --- Sorry, Claudio, my bad. I still had remnants of GIRAPH-64. It's fine and passes local unittests. Export Worker's Context/State to vertices through pre/post/Application/Superstep Key: GIRAPH-47 URL: https://issues.apache.org/jira/browse/GIRAPH-47 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-47.diff, GIRAPH-47.diff It would be quite useful for vertices to reach some worker-related information stored i.e. in the GraphState class. This information could be exported as a parameter to pre/post/Application/Superstep like this: public void preApplication(Configurable workerObject); public void postApplication(Configurable workerObject); public void preSuperstep(Configurable workerObject); public void postSuperstep(Configurable workerObject); public Configurable getWorkerObject(); Another possibility is to add a Context inner class to BasicVertex to store this information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-47) Export Worker's Context/State to vertices through pre/post/Application/Superstep
[ https://issues.apache.org/jira/browse/GIRAPH-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145716#comment-13145716 ] jirapos...@reviews.apache.org commented on GIRAPH-47: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2746/#review3082 --- Claudio, really nice stuff here. Most of my comments are related to indenting. But otherwise, this is a lot better IMO. Please take a look at CODE_CONVENTIONS and fix accordingly. While the official policy is 2 space, at this time, for the 4 space indented files, please keep to 4 spaces for consistency. We will transition everything over at some point. New files can be 2 space (new convention) if desired. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6885 This doesn't need to be static anymore. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6870 Indenting should be 8 spaces. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6873 extra line. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6874 extra line. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6875 4 spaces. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6871 4 spaces indenting. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6872 4 spaces indenting. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java https://reviews.apache.org/r/2746/#comment6876 Align to GiraphJob.WORKER_CONTEXT_CLASS http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java https://reviews.apache.org/r/2746/#comment6877 VERTEX_COUNT shouldn't be capitalized. All caps should be reserved for only static values. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java https://reviews.apache.org/r/2746/#comment6878 EDGE_COUNT shouldn't be capitalized. All caps should be reserved for only static values. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java https://reviews.apache.org/r/2746/#comment6887 These no longer need to be static anymore, could be private variables that have public accessor method. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java https://reviews.apache.org/r/2746/#comment6879 Might want to add a comment about this example. I.e. /** * Fully runnable example of how to * emit worker data to HDFS during a graph * computation. */ http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java https://reviews.apache.org/r/2746/#comment6880 extra line. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java https://reviews.apache.org/r/2746/#comment6881 Awesome, I hated this. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java https://reviews.apache.org/r/2746/#comment6882 indenting. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/DefaultWorkerContext.java https://reviews.apache.org/r/2746/#comment6883 extra line. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/WorkerContext.java https://reviews.apache.org/r/2746/#comment6884 Other javadoc has lines in between comment and params (i.e. * superstep starts. * * @throws IllegalAccessException - Avery On 2011-11-07 19:09:08, Avery Ching wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq.
[jira] [Commented] (GIRAPH-47) Export Worker's Context/State to vertices through pre/post/Application/Superstep
[ https://issues.apache.org/jira/browse/GIRAPH-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145754#comment-13145754 ] jirapos...@reviews.apache.org commented on GIRAPH-47: - bq. On 2011-11-07 19:12:55, Avery Ching wrote: bq. Claudio, really nice stuff here. Most of my comments are related to indenting. But otherwise, this is a lot better IMO. Please take a look at CODE_CONVENTIONS and fix accordingly. While the official policy is 2 space, at this time, for the 4 space indented files, please keep to 4 spaces for consistency. We will transition everything over at some point. New files can be 2 space (new convention) if desired. Ok, still have to understand a bit the code conventions. Trying to stick to them. Maybe an Eclipse format conf file would help? Could you share yours, if you have one? bq. On 2011-11-07 19:12:55, Avery Ching wrote: bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java, lines 91-92 bq. https://reviews.apache.org/r/2746/diff/1/?file=56634#file56634line91 bq. bq. These no longer need to be static anymore, could be private variables that have public accessor method. Not sure we can do this. How will tests get to their values. Can't access those members if not static. bq. On 2011-11-07 19:12:55, Avery Ching wrote: bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java, lines 250-251 bq. https://reviews.apache.org/r/2746/diff/1/?file=56632#file56632line250 bq. bq. Align to GiraphJob.WORKER_CONTEXT_CLASS What do you mean? I aligned to the example, all classes are set with .setClass() there. Fixing the whole thing. bq. On 2011-11-07 19:12:55, Avery Ching wrote: bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java, line 150 bq. https://reviews.apache.org/r/2746/diff/1/?file=56632#file56632line150 bq. bq. This doesn't need to be static anymore. Can't make it non static. Won't be able to read from tests. - Claudio --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2746/#review3082 --- On 2011-11-07 19:09:08, Avery Ching wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2746/ bq. --- bq. bq. (Updated 2011-11-07 19:09:08) bq. bq. bq. Review request for giraph. bq. bq. bq. Summary bq. --- bq. bq. Claudio's patch for GIRAPH-47. bq. bq. bq. This addresses bug GIRAPH-47. bq. https://issues.apache.org/jira/browse/GIRAPH-47 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/benchmark/RandomMessageBenchmark.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedService.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleCheckpointVertex.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleMutateGraphVertex.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimplePageRankVertex.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/examples/SimpleVertexWithWorkerContext.java PRE-CREATION bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BasicVertex.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspService.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/BspUtils.java 1198865 bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/DefaultWorkerContext.java PRE-CREATION bq. http://svn.apache.org/repos/asf/incubator/giraph/trunk/src/main/java/org/apache/giraph/graph/GiraphJob.java 1198865
FOSDEM
Hello list, I was thinking about submitting for a talk at fosdem with topic Pregel Giraph. Am I overlapping with somebody else? Best, Claudio -- Claudio Martella claudio.marte...@gmail.com
[jira] [Commented] (GIRAPH-47) Export Worker's Context/State to vertices through pre/post/Application/Superstep
[ https://issues.apache.org/jira/browse/GIRAPH-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145868#comment-13145868 ] Avery Ching commented on GIRAPH-47: --- +1 Thanks Claudio. I'm going to commit on your behalf. I made a few small changes to fix the code conventions. I also fixed a bug in the distributed unittests (WorkerContext not set for TestAutoCheckpoint). The local unittests can't catch that one. Next time, you'll want to run against a real Hadoop instance to catch problems (i.e. mvn test -Dprop.mapred.job.tracker=localhost:50300). You'll want to replace 'localhost:50300' with your actual Hadoop jobtracker and port settings. Export Worker's Context/State to vertices through pre/post/Application/Superstep Key: GIRAPH-47 URL: https://issues.apache.org/jira/browse/GIRAPH-47 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-47.diff, GIRAPH-47.diff, GIRAPH-47.diff It would be quite useful for vertices to reach some worker-related information stored i.e. in the GraphState class. This information could be exported as a parameter to pre/post/Application/Superstep like this: public void preApplication(Configurable workerObject); public void postApplication(Configurable workerObject); public void preSuperstep(Configurable workerObject); public void postSuperstep(Configurable workerObject); public Configurable getWorkerObject(); Another possibility is to add a Context inner class to BasicVertex to store this information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-47) Export Worker's Context/State to vertices through pre/post/Application/Superstep
[ https://issues.apache.org/jira/browse/GIRAPH-47?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13145879#comment-13145879 ] Avery Ching commented on GIRAPH-47: --- I filed issue https://issues.apache.org/jira/browse/INFRA-4097. Something is weird with hudson, cannot find mvn. https://builds.apache.org/job/Giraph-trunk-Commit/24/console snip [Giraph-trunk-Commit] $ /bin/bash -x /tmp/hudson6174557603739055037.sh + export JAVA_HOME=/home/jenkins/tools/java/latest + JAVA_HOME=/home/jenkins/tools/java/latest + export MAVEN_HOME=/home/jenkins/tools/maven/latest + MAVEN_HOME=/home/jenkins/tools/maven/latest + export PATH=/home/jenkins/tools/java/latest/bin:/home/jenkins/tools/maven/latest/bin:/home/hudson/tools/java/latest1.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/home/hudson/.rvm/bin + PATH=/home/jenkins/tools/java/latest/bin:/home/jenkins/tools/maven/latest/bin:/home/hudson/tools/java/latest1.6/bin:/usr/local/bin:/usr/bin:/bin:/usr/bin/X11:/usr/games:/home/hudson/.rvm/bin + which java /home/hudson/tools/java/latest1.6/bin/java + cd /home/hudson/hudson-slave/workspace/Giraph-trunk-Commit/trunk + /home/jenkins/tools/maven/latest/bin/mvn clean compile test /tmp/hudson6174557603739055037.sh: line 9: /home/jenkins/tools/maven/latest/bin/mvn: No such file or directory Build step 'Execute shell' marked build as failure Updating GIRAPH-47 Recording test results Finished: FAILURE Export Worker's Context/State to vertices through pre/post/Application/Superstep Key: GIRAPH-47 URL: https://issues.apache.org/jira/browse/GIRAPH-47 Project: Giraph Issue Type: Improvement Components: graph Affects Versions: 0.70.0 Reporter: Claudio Martella Assignee: Claudio Martella Attachments: GIRAPH-47.diff, GIRAPH-47.diff, GIRAPH-47.diff It would be quite useful for vertices to reach some worker-related information stored i.e. in the GraphState class. This information could be exported as a parameter to pre/post/Application/Superstep like this: public void preApplication(Configurable workerObject); public void postApplication(Configurable workerObject); public void preSuperstep(Configurable workerObject); public void postSuperstep(Configurable workerObject); public Configurable getWorkerObject(); Another possibility is to add a Context inner class to BasicVertex to store this information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-64) Create VertexRunner to make it easier to run users' computations
[
https://issues.apache.org/jira/browse/GIRAPH-64?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13146023#comment-13146023
]
Hyunsik Choi commented on GIRAPH-64:
In my case, 'mvn package' is ok, but 'mvn assembly:assembly' incurs the error
as I mentioned above.
{code}
hyunsik@code:~$ mvn --version
Apache Maven 3.0.3 (r1075438; 2011-03-01 02:31:09+0900)
Maven home: /home/hyunsik/Local/maven-3
Java version: 1.6.0_26, vendor: Sun Microsystems Inc.
Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre
Default locale: en_US, platform encoding: UTF-8
OS name: linux, version: 3.0.0-12-generic, arch: amd64, family: unix
{code}
Create VertexRunner to make it easier to run users' computations
Key: GIRAPH-64
URL: https://issues.apache.org/jira/browse/GIRAPH-64
Project: Giraph
Issue Type: New Feature
Reporter: Jakob Homan
Assignee: Jakob Homan
Attachments: GIRAPH-64.patch
Currently, if a user wants to implement a Giraph algorithm by extending
{{Vertex}} they must also write all the boilerplate around the {{Tool}}
interface and bundle it with the Giraph jar (or get Giraph on the classpath
and playing nice with the implementation). For example, what is included in
the PageRankBenchmark and what Kohei has done:
https://github.com/smly/java-Giraph-LabelPropagation It would be better if
we had perhaps a Vertex implementation to be subclassed that already had all
the standard Tooling included such that all one had to run would be (assuming
the Giraph jar was already on the classpath):
{noformat}hadoop jar my-awesome-vertex.jar my.awesome.vertex -i jazz_input -o
jazz_output -if org.apache.giraph.lib.in.text.adjacency-list.LongDoubleDouble
-of org.apache.giraph.lib.out.text.adjacency-list.LongDoubleDouble{noformat}
This wouldn't work with every algorithm, but would be useful in a large
number of cases.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
