Re: Getting Started Code
On 20 Apr 2012, at 17:27, Etienne Dumoulin wrote: > Like it is a bit difficult to start from scratch a first time, > I copied/pasted the getting started code. > > I do not understand why there is 3 template classes when > TextVertexInputFormat has 4 > (from the javadoc and eclipse). > > For example: > "public static class SimpleShortestPathsVertexInputFormat extends > TextVertexInputFormat" > > Is there something I miss? Personally, I dont think that the code to which the getting started guide refers, is the best place to start *understanding* Giraph. But its the best code to execute in order to see if your Hadoop setup actually works for Giraph. A better example for understanding Giraph IMHO is the ConnectedComponentsVertexTest class (in the test directory of the code). It shows that you need to implement a Vertex class, an TextInputFormat and a TextOutputFormat class in order to define your own job. Then you can use InternalVertexRunner.run() in order to test your code inside of Giraph, and figure out Hadoop later. I was puzzled about your "3 templates versus 4" error, but then I tried it out in Eclipse. If you are also using eclipse, then I know what you mean ;) If you mouse over e.g. SimpleShortesPathsVertex, then eclipse will display a tool-tip. However, thats just the top part of the javadoc with 3 template parameters, you need to scroll down, in order to see all 4 elements of the template signature. To do that, you can either press F2, to see the scroll bar or resize the window. Or you click on the class, and then you can read the source of the javadoc. However, keep in mind, that some specific classes which are used in the TextInputFormat and the TextOutputFormat have template signatures which really only have 2 or 3 elements. As a second thing, I would look at the SimpleShortestPathVertexTest class (again in the test directory). Then you can look at the PageRankeBenchmark class in giraph.benchmark. Let me/us know if you have more questions.
Getting Started Code
Hi Guys, I am just trying to create my first giraph code today. Like it is a bit difficult to start from scratch a first time, I copied/pasted the getting started code. I do not understand why there is 3 template classes when TextVertexInputFormat has 4 (from the javadoc and eclipse). For example: "public static class SimpleShortestPathsVertexInputFormat extends TextVertexInputFormat" Is there something I miss? Also, what I would like to do is creating a code for very general graph measurements in a weighted graphs (no weight on the vertexes), such as degree, distances, centrality. It is a bit hard to start only with the javadoc :/. Could you gives me some clues, what are the classes to implements for example? I tried to move on with the getting started but the class to implement for 3 and 4 templates are completely different and I suppose there is a simpler way. Thanks, Étienne
Re: Getting started
Hey Shunak- Sorry, you've caught us with the place in a bit of a mess. Until GIRAPH-136 (which I just committed), it wasn't possible to run bin/giraph from a fresh checkout of the source. bin/giraph was assumed to be run from the result of unpackaging of giraph-.tar.gz. With GIRAPH-136, after running mvn:jar, it should be possible to run bin/giraph (assuming a valid Hadoop install). Please let us know as you find these issues; we're working on improving the user experience, but we're still at an early stage. -Jakob On Thu, Feb 2, 2012 at 6:06 PM, Shaunak Kashyap wrote: > I'm sorry, but I don't follow and am a bit confused. Is there an example I > could look at? > > What I'm trying to achieve is using the giraph-*.jar as a dependency of my > project. In my project I would like to define a class similar to > org.apache.giraph.examples.SimpleShortestPathsVertex. Then I would like to > create a jar from my project and use that with "hadoop jar ..." or > "bin/giraph ...". > > Hope that made sense. Thank you, > > Shaunak > > > On Thu, Feb 2, 2012 at 1:40 PM, David Garcia > wrote: >> >> Before you run the jar, run mvn clean package from trunk. After that, put >> the snapshot (or the with-dependencies) jar on the class path for your >> example. It should work after that. >> >> From: Shaunak Kashyap >> Reply-To: "giraph-user@incubator.apache.org" >> >> Date: Wed, 1 Feb 2012 20:44:09 -0600 >> To: "giraph-user@incubator.apache.org" >> Subject: Re: Getting started >> >> Thanks, Jakob. >> >> I downloaded the source >> from https://svn.apache.org/repos/asf/incubator/giraph/trunk, built it using >> "mvn compile", then issued this command from the base directory (where >> pom.xml lives): >> >> $ ./bin/giraph ../giraph-example-shortest-path-with-libs.jar >> com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >> >> And I got back this error: >> >> Can't find Giraph jar. >> >> Digging into the contents of ./bin/giraph, it looks like its expecting the >> Giraph JAR in lib/ whereas "mvn compile" places it in target/. >> >> Perhaps I shouldn't be compiling from source but using the "regular Giraph >> .tar.gz" you mentioned in your email. I couldn't find it referenced from >> anywhere on http://incubator.apache.org/giraph/ (but maybe I haven't looked >> hard enough). >> >> Thanks, >> >> Shaunak >> >> On Wed, Feb 1, 2012 at 11:17 AM, Jakob Homan wrote: >>> >>> We need to update the wiki; it's out of date. Now we have bin/giraph >>> to run user jars. Once you have the regular Giraph .tar.gz unpacked >>> on your machine and your vertex jar somewhere: >>> bin/giraph >> directories, formats, etc...> >>> >>> >>> On Wed, Feb 1, 2012 at 11:09 AM, Shaunak Kashyap >>> wrote: >>> > Hi, >>> > >>> > I was able to run the shortest paths example as described >>> > >>> > here: https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example. >>> > >>> > After this I tried to create my own JAR containing a class similar >>> > to SimpleShortestPathsVertex and a "lib" directory containing the >>> > giraph-*.jar (obtained from the target/ directory after building from >>> > source) and other dependent JARs. When I tried to run my JAR as a >>> > hadoop >>> > job, I get the following errors: >>> > >>> > $ hadoop jar giraph-example-shortest-path-with-libs.jar >>> > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >>> > shortestPathsInputGraph shortestPathsOutputGraph 0 3 >>> > >>> > 12/02/01 11:05:01 INFO mapred.JobClient: Running job: >>> > job_20161517_1471 >>> > 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% >>> > 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : >>> > attempt_20161517_1471_m_01_0, Status : FAILED >>> > java.lang.IllegalStateException: run: Caught an unrecoverable exception >>> > java.lang.RuntimeException: java.lang.ClassNotFoundException: >>> > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >>> > at >>> > org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) >>> > at >>> > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) >>> >
Re: Getting started
I'm sorry, but I don't follow and am a bit confused. Is there an example I could look at? What I'm trying to achieve is using the giraph-*.jar as a dependency of my project. In my project I would like to define a class similar to org.apache.giraph.examples.SimpleShortestPathsVertex. Then I would like to create a jar from my project and use that with "hadoop jar ..." or "bin/giraph ...". Hope that made sense. Thank you, Shaunak On Thu, Feb 2, 2012 at 1:40 PM, David Garcia wrote: > Before you run the jar, run mvn clean package from trunk. After that, put > the snapshot (or the with-dependencies) jar on the class path for your > example. It should work after that. > > From: Shaunak Kashyap > Reply-To: "giraph-user@incubator.apache.org" < > giraph-user@incubator.apache.org> > Date: Wed, 1 Feb 2012 20:44:09 -0600 > To: "giraph-user@incubator.apache.org" > Subject: Re: Getting started > > Thanks, Jakob. > > I downloaded the source from > https://svn.apache.org/repos/asf/incubator/giraph/trunk, built it using > "mvn compile", then issued this command from the base directory (where > pom.xml lives): > > $ ./bin/giraph ../giraph-example-shortest-path-with-libs.jar > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > > And I got back this error: > > Can't find Giraph jar. > > Digging into the contents of ./bin/giraph, it looks like its expecting the > Giraph JAR in lib/ whereas "mvn compile" places it in target/. > > Perhaps I shouldn't be compiling from source but using the "regular Giraph > .tar.gz" you mentioned in your email. I couldn't find it referenced from > anywhere on http://incubator.apache.org/giraph/ (but maybe I haven't > looked hard enough). > > Thanks, > > Shaunak > > On Wed, Feb 1, 2012 at 11:17 AM, Jakob Homan wrote: > >> We need to update the wiki; it's out of date. Now we have bin/giraph >> to run user jars. Once you have the regular Giraph .tar.gz unpacked >> on your machine and your vertex jar somewhere: >> bin/giraph > directories, formats, etc...> >> >> >> On Wed, Feb 1, 2012 at 11:09 AM, Shaunak Kashyap >> wrote: >> > Hi, >> > >> > I was able to run the shortest paths example as described >> > here: >> https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example >> . >> > >> > After this I tried to create my own JAR containing a class similar >> > to SimpleShortestPathsVertex and a "lib" directory containing the >> > giraph-*.jar (obtained from the target/ directory after building from >> > source) and other dependent JARs. When I tried to run my JAR as a hadoop >> > job, I get the following errors: >> > >> > $ hadoop jar giraph-example-shortest-path-with-libs.jar >> > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >> > shortestPathsInputGraph shortestPathsOutputGraph 0 3 >> > >> > 12/02/01 11:05:01 INFO mapred.JobClient: Running job: >> job_20161517_1471 >> > 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% >> > 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : >> > attempt_20161517_1471_m_01_0, Status : FAILED >> > java.lang.IllegalStateException: run: Caught an unrecoverable exception >> > java.lang.RuntimeException: java.lang.ClassNotFoundException: >> > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >> > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) >> > at >> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) >> > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) >> > at java.security.AccessController.doPrivileged(Native Method) >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> > at >> > >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) >> > at org.apache.hadoop.mapred.Child.main(Child.java:264) >> > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: >> > java.lang.ClassNotFoundException: >> > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex >> > at >> > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028) >> > at >> > org.apache.giraph.graph.BspUtils.getVertexClass(BspUtils.java:299) >> > >> > When I checked, the class
Re: Getting started
Before you run the jar, run mvn clean package from trunk. After that, put the snapshot (or the with-dependencies) jar on the class path for your example. It should work after that. From: Shaunak Kashyap mailto:ycombina...@gmail.com>> Reply-To: "giraph-user@incubator.apache.org<mailto:giraph-user@incubator.apache.org>" mailto:giraph-user@incubator.apache.org>> Date: Wed, 1 Feb 2012 20:44:09 -0600 To: "giraph-user@incubator.apache.org<mailto:giraph-user@incubator.apache.org>" mailto:giraph-user@incubator.apache.org>> Subject: Re: Getting started Thanks, Jakob. I downloaded the source from https://svn.apache.org/repos/asf/incubator/giraph/trunk, built it using "mvn compile", then issued this command from the base directory (where pom.xml lives): $ ./bin/giraph ../giraph-example-shortest-path-with-libs.jar com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex And I got back this error: Can't find Giraph jar. Digging into the contents of ./bin/giraph, it looks like its expecting the Giraph JAR in lib/ whereas "mvn compile" places it in target/. Perhaps I shouldn't be compiling from source but using the "regular Giraph .tar.gz" you mentioned in your email. I couldn't find it referenced from anywhere on http://incubator.apache.org/giraph/ (but maybe I haven't looked hard enough). Thanks, Shaunak On Wed, Feb 1, 2012 at 11:17 AM, Jakob Homan mailto:jgho...@gmail.com>> wrote: We need to update the wiki; it's out of date. Now we have bin/giraph to run user jars. Once you have the regular Giraph .tar.gz unpacked on your machine and your vertex jar somewhere: bin/giraph On Wed, Feb 1, 2012 at 11:09 AM, Shaunak Kashyap mailto:ycombina...@gmail.com>> wrote: > Hi, > > I was able to run the shortest paths example as described > here: > https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example. > > After this I tried to create my own JAR containing a class similar > to SimpleShortestPathsVertex and a "lib" directory containing the > giraph-*.jar (obtained from the target/ directory after building from > source) and other dependent JARs. When I tried to run my JAR as a hadoop > job, I get the following errors: > > $ hadoop jar giraph-example-shortest-path-with-libs.jar > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > shortestPathsInputGraph shortestPathsOutputGraph 0 3 > > 12/02/01 11:05:01 INFO mapred.JobClient: Running job: job_20161517_1471 > 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% > 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : > attempt_20161517_1471_m_01_0, Status : FAILED > java.lang.IllegalStateException: run: Caught an unrecoverable exception > java.lang.RuntimeException: java.lang.ClassNotFoundException: > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028) > at > org.apache.giraph.graph.BspUtils.getVertexClass(BspUtils.java:299) > > When I checked, the class does exist in my JAR, in the right path: > > $ jar tvf giraph-example-shortest-path-with-libs.jar | grep > SkyboxShortestPathsVertex > 6029 Tue Jan 31 13:22:50 PST 2012 > com/skyboximaging/janus/giraph/example/SkyboxShortestPathsVertex.class > > Maybe this is yet another Hadoop issue (i.e. not specific to Giraph) but I'd > appreciate any help with it. > > Thank you, > > Shaunak > > On Wed, Dec 14, 2011 at 6:57 AM, Shaunak Kashyap > mailto:ycombina...@gmail.com>> > wrote: >> >> Hi Avery, >> >> You're right; my error turned out to be a Hadoop issue which I have >> worked around now. Thanks for the official documentation link. >> >> Shaunak >> >> On Tue, Dec 13, 2011 at 5:07 PM, Avery Ching >> mailto:ach...@apache.org>> wrote: >> > Hi Shaunak, >> > >> > The official GIRAPH documentation is lo
Re: Getting started
Thanks, Jakob. I downloaded the source from https://svn.apache.org/repos/asf/incubator/giraph/trunk, built it using "mvn compile", then issued this command from the base directory (where pom.xml lives): $ ./bin/giraph ../giraph-example-shortest-path-with-libs.jar com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex And I got back this error: Can't find Giraph jar. Digging into the contents of ./bin/giraph, it looks like its expecting the Giraph JAR in lib/ whereas "mvn compile" places it in target/. Perhaps I shouldn't be compiling from source but using the "regular Giraph .tar.gz" you mentioned in your email. I couldn't find it referenced from anywhere on http://incubator.apache.org/giraph/ (but maybe I haven't looked hard enough). Thanks, Shaunak On Wed, Feb 1, 2012 at 11:17 AM, Jakob Homan wrote: > We need to update the wiki; it's out of date. Now we have bin/giraph > to run user jars. Once you have the regular Giraph .tar.gz unpacked > on your machine and your vertex jar somewhere: > bin/giraphdirectories, formats, etc...> > > > On Wed, Feb 1, 2012 at 11:09 AM, Shaunak Kashyap > wrote: > > Hi, > > > > I was able to run the shortest paths example as described > > here: > https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example. > > > > After this I tried to create my own JAR containing a class similar > > to SimpleShortestPathsVertex and a "lib" directory containing the > > giraph-*.jar (obtained from the target/ directory after building from > > source) and other dependent JARs. When I tried to run my JAR as a hadoop > > job, I get the following errors: > > > > $ hadoop jar giraph-example-shortest-path-with-libs.jar > > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > > shortestPathsInputGraph shortestPathsOutputGraph 0 3 > > > > 12/02/01 11:05:01 INFO mapred.JobClient: Running job: > job_20161517_1471 > > 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% > > 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : > > attempt_20161517_1471_m_01_0, Status : FAILED > > java.lang.IllegalStateException: run: Caught an unrecoverable exception > > java.lang.RuntimeException: java.lang.ClassNotFoundException: > > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) > > at > org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at > > > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > > at org.apache.hadoop.mapred.Child.main(Child.java:264) > > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > > java.lang.ClassNotFoundException: > > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > > at > > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028) > > at > > org.apache.giraph.graph.BspUtils.getVertexClass(BspUtils.java:299) > > > > When I checked, the class does exist in my JAR, in the right path: > > > > $ jar tvf giraph-example-shortest-path-with-libs.jar | grep > > SkyboxShortestPathsVertex > > 6029 Tue Jan 31 13:22:50 PST 2012 > > com/skyboximaging/janus/giraph/example/SkyboxShortestPathsVertex.class > > > > Maybe this is yet another Hadoop issue (i.e. not specific to Giraph) but > I'd > > appreciate any help with it. > > > > Thank you, > > > > Shaunak > > > > On Wed, Dec 14, 2011 at 6:57 AM, Shaunak Kashyap > > wrote: > >> > >> Hi Avery, > >> > >> You're right; my error turned out to be a Hadoop issue which I have > >> worked around now. Thanks for the official documentation link. > >> > >> Shaunak > >> > >> On Tue, Dec 13, 2011 at 5:07 PM, Avery Ching wrote: > >> > Hi Shaunak, > >> > > >> > The official GIRAPH documentation is located at > >> > https://cwiki.apache.org/confluence/display/GIRAPH/Index . That > being > >> > said, > >> > your error looks like a Hadoop issue. Are you able to run normal > Hadoop > >> > example jobs? > >> > > >> > Thanks, > >> > > >> > Avery > >> > > >> > > >> > On 12/13/11 9:27 AM, Shaunak Kashyap wrote: > >> >> > >> >> Hi, > >> >> > >> >> Maybe this page - > >> >> https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is > outdated > >> >> but I tried following the instructions there and am getting the > >> >> following error: > >> >> > >> >> shaunak@hadoop-1:~/Giraph$ hadoop jar > >> >> target/giraph-0.70-jar-with-dependencies.jar > >> >> org.apache.giraph.benchmark.PageRankBenchmark -h > >> >> 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: > >> >> javax.xml.parsers.ParserConfigurationException: Feature > >> >> 'http://apache.org/xml/features/xinclude' is not recogn
Re: Getting started
We need to update the wiki; it's out of date. Now we have bin/giraph to run user jars. Once you have the regular Giraph .tar.gz unpacked on your machine and your vertex jar somewhere: bin/giraph On Wed, Feb 1, 2012 at 11:09 AM, Shaunak Kashyap wrote: > Hi, > > I was able to run the shortest paths example as described > here: https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example. > > After this I tried to create my own JAR containing a class similar > to SimpleShortestPathsVertex and a "lib" directory containing the > giraph-*.jar (obtained from the target/ directory after building from > source) and other dependent JARs. When I tried to run my JAR as a hadoop > job, I get the following errors: > > $ hadoop jar giraph-example-shortest-path-with-libs.jar > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > shortestPathsInputGraph shortestPathsOutputGraph 0 3 > > 12/02/01 11:05:01 INFO mapred.JobClient: Running job: job_20161517_1471 > 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% > 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : > attempt_20161517_1471_m_01_0, Status : FAILED > java.lang.IllegalStateException: run: Caught an unrecoverable exception > java.lang.RuntimeException: java.lang.ClassNotFoundException: > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) > at org.apache.hadoop.mapred.Child$4.run(Child.java:270) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) > at org.apache.hadoop.mapred.Child.main(Child.java:264) > Caused by: java.lang.RuntimeException: java.lang.RuntimeException: > java.lang.ClassNotFoundException: > com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex > at > org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028) > at > org.apache.giraph.graph.BspUtils.getVertexClass(BspUtils.java:299) > > When I checked, the class does exist in my JAR, in the right path: > > $ jar tvf giraph-example-shortest-path-with-libs.jar | grep > SkyboxShortestPathsVertex > 6029 Tue Jan 31 13:22:50 PST 2012 > com/skyboximaging/janus/giraph/example/SkyboxShortestPathsVertex.class > > Maybe this is yet another Hadoop issue (i.e. not specific to Giraph) but I'd > appreciate any help with it. > > Thank you, > > Shaunak > > On Wed, Dec 14, 2011 at 6:57 AM, Shaunak Kashyap > wrote: >> >> Hi Avery, >> >> You're right; my error turned out to be a Hadoop issue which I have >> worked around now. Thanks for the official documentation link. >> >> Shaunak >> >> On Tue, Dec 13, 2011 at 5:07 PM, Avery Ching wrote: >> > Hi Shaunak, >> > >> > The official GIRAPH documentation is located at >> > https://cwiki.apache.org/confluence/display/GIRAPH/Index . That being >> > said, >> > your error looks like a Hadoop issue. Are you able to run normal Hadoop >> > example jobs? >> > >> > Thanks, >> > >> > Avery >> > >> > >> > On 12/13/11 9:27 AM, Shaunak Kashyap wrote: >> >> >> >> Hi, >> >> >> >> Maybe this page - >> >> https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is outdated >> >> but I tried following the instructions there and am getting the >> >> following error: >> >> >> >> shaunak@hadoop-1:~/Giraph$ hadoop jar >> >> target/giraph-0.70-jar-with-dependencies.jar >> >> org.apache.giraph.benchmark.PageRankBenchmark -h >> >> 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: >> >> javax.xml.parsers.ParserConfigurationException: Feature >> >> 'http://apache.org/xml/features/xinclude' is not recognized. >> >> Exception in thread "main" java.lang.RuntimeException: >> >> javax.xml.parsers.ParserConfigurationException: Feature >> >> 'http://apache.org/xml/features/xinclude' is not recognized. >> >> at >> >> >> >> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1396) >> >> at >> >> >> >> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251) >> >> at >> >> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192) >> >> at >> >> org.apache.hadoop.conf.Configuration.get(Configuration.java:415) >> >> at org.apache.hadoop.util.RunJar.main(RunJar.java:143) >> >> Caused by: javax.xml.parsers.ParserConfigurationException: Feature >> >> 'http://apache.org/xml/features/xinclude' is not recognized. >> >> at >> >> >> >> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown >> >> Source) >> >> at >> >> >> >> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1282) >> >> ... 4 more >> >> >> >> What am I missing? >>
Re: Getting started
Hi, I was able to run the shortest paths example as described here: https://cwiki.apache.org/confluence/display/GIRAPH/Shortest+Paths+Example. After this I tried to create my own JAR containing a class similar to SimpleShortestPathsVertex and a "lib" directory containing the giraph-*.jar (obtained from the target/ directory after building from source) and other dependent JARs. When I tried to run my JAR as a hadoop job, I get the following errors: $ hadoop jar giraph-example-shortest-path-with-libs.jar com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex shortestPathsInputGraph shortestPathsOutputGraph 0 3 12/02/01 11:05:01 INFO mapred.JobClient: Running job: job_20161517_1471 12/02/01 11:05:02 INFO mapred.JobClient: map 0% reduce 0% 12/02/01 11:05:07 INFO mapred.JobClient: Task Id : attempt_20161517_1471_m_01_0, Status : FAILED java.lang.IllegalStateException: run: Caught an unrecoverable exception java.lang.RuntimeException: java.lang.ClassNotFoundException: com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:650) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:647) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: com.skyboximaging.janus.giraph.example.SkyboxShortestPathsVertex at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1028) at org.apache.giraph.graph.BspUtils.getVertexClass(BspUtils.java:299) When I checked, the class does exist in my JAR, in the right path: $ jar tvf giraph-example-shortest-path-with-libs.jar | grep SkyboxShortestPathsVertex 6029 Tue Jan 31 13:22:50 PST 2012 com/skyboximaging/janus/giraph/example/SkyboxShortestPathsVertex.class Maybe this is yet another Hadoop issue (i.e. not specific to Giraph) but I'd appreciate any help with it. Thank you, Shaunak On Wed, Dec 14, 2011 at 6:57 AM, Shaunak Kashyap wrote: > Hi Avery, > > You're right; my error turned out to be a Hadoop issue which I have > worked around now. Thanks for the official documentation link. > > Shaunak > > On Tue, Dec 13, 2011 at 5:07 PM, Avery Ching wrote: > > Hi Shaunak, > > > > The official GIRAPH documentation is located at > > https://cwiki.apache.org/confluence/display/GIRAPH/Index . That being > said, > > your error looks like a Hadoop issue. Are you able to run normal Hadoop > > example jobs? > > > > Thanks, > > > > Avery > > > > > > On 12/13/11 9:27 AM, Shaunak Kashyap wrote: > >> > >> Hi, > >> > >> Maybe this page - > >> https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is outdated > >> but I tried following the instructions there and am getting the > >> following error: > >> > >> shaunak@hadoop-1:~/Giraph$ hadoop jar > >> target/giraph-0.70-jar-with-dependencies.jar > >> org.apache.giraph.benchmark.PageRankBenchmark -h > >> 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: > >> javax.xml.parsers.ParserConfigurationException: Feature > >> 'http://apache.org/xml/features/xinclude' is not recognized. > >> Exception in thread "main" java.lang.RuntimeException: > >> javax.xml.parsers.ParserConfigurationException: Feature > >> 'http://apache.org/xml/features/xinclude' is not recognized. > >> at > >> > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1396) > >> at > >> > org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251) > >> at > >> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192) > >> at > >> org.apache.hadoop.conf.Configuration.get(Configuration.java:415) > >> at org.apache.hadoop.util.RunJar.main(RunJar.java:143) > >> Caused by: javax.xml.parsers.ParserConfigurationException: Feature > >> 'http://apache.org/xml/features/xinclude' is not recognized. > >> at > >> > org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown > >> Source) > >> at > >> > org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1282) > >> ... 4 more > >> > >> What am I missing? > >> > >> Thank you, > >> > >> Shaunak > >> > > > > > > -- > "Now the hardness of this world slowly grinds your dreams away / > Makin' a fool's joke out of the promises we make" --- Bruce > Springsteen, "Blood Brothers" > -- "Now the hardness of this world slowly grinds your dreams away / Makin' a fool's joke out of the promises we make" --- Bruce Springsteen, "Blood Brothers"
Re: Getting started
Hi Avery, You're right; my error turned out to be a Hadoop issue which I have worked around now. Thanks for the official documentation link. Shaunak On Tue, Dec 13, 2011 at 5:07 PM, Avery Ching wrote: > Hi Shaunak, > > The official GIRAPH documentation is located at > https://cwiki.apache.org/confluence/display/GIRAPH/Index . That being said, > your error looks like a Hadoop issue. Are you able to run normal Hadoop > example jobs? > > Thanks, > > Avery > > > On 12/13/11 9:27 AM, Shaunak Kashyap wrote: >> >> Hi, >> >> Maybe this page - >> https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is outdated >> but I tried following the instructions there and am getting the >> following error: >> >> shaunak@hadoop-1:~/Giraph$ hadoop jar >> target/giraph-0.70-jar-with-dependencies.jar >> org.apache.giraph.benchmark.PageRankBenchmark -h >> 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: >> javax.xml.parsers.ParserConfigurationException: Feature >> 'http://apache.org/xml/features/xinclude' is not recognized. >> Exception in thread "main" java.lang.RuntimeException: >> javax.xml.parsers.ParserConfigurationException: Feature >> 'http://apache.org/xml/features/xinclude' is not recognized. >> at >> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1396) >> at >> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251) >> at >> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192) >> at >> org.apache.hadoop.conf.Configuration.get(Configuration.java:415) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:143) >> Caused by: javax.xml.parsers.ParserConfigurationException: Feature >> 'http://apache.org/xml/features/xinclude' is not recognized. >> at >> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown >> Source) >> at >> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1282) >> ... 4 more >> >> What am I missing? >> >> Thank you, >> >> Shaunak >> > -- "Now the hardness of this world slowly grinds your dreams away / Makin' a fool's joke out of the promises we make" --- Bruce Springsteen, "Blood Brothers"
Re: Getting started
Hi Shaunak, The official GIRAPH documentation is located at https://cwiki.apache.org/confluence/display/GIRAPH/Index . That being said, your error looks like a Hadoop issue. Are you able to run normal Hadoop example jobs? Thanks, Avery On 12/13/11 9:27 AM, Shaunak Kashyap wrote: Hi, Maybe this page - https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is outdated but I tried following the instructions there and am getting the following error: shaunak@hadoop-1:~/Giraph$ hadoop jar target/giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -h 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1396) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192) at org.apache.hadoop.conf.Configuration.get(Configuration.java:415) at org.apache.hadoop.util.RunJar.main(RunJar.java:143) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1282) ... 4 more What am I missing? Thank you, Shaunak
Getting started
Hi, Maybe this page - https://github.com/aching/Giraph/wiki/Quick-Start-Guide - is outdated but I tried following the instructions there and am getting the following error: shaunak@hadoop-1:~/Giraph$ hadoop jar target/giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -h 11/12/13 09:25:08 FATAL conf.Configuration: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. Exception in thread "main" java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1396) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1251) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1192) at org.apache.hadoop.conf.Configuration.get(Configuration.java:415) at org.apache.hadoop.util.RunJar.main(RunJar.java:143) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1282) ... 4 more What am I missing? Thank you, Shaunak -- "Now the hardness of this world slowly grinds your dreams away / Makin' a fool's joke out of the promises we make" --- Bruce Springsteen, "Blood Brothers"
Re: Can't get Getting Started example to work
Glad to hear that got resolved Kyle. Avery On 9/7/11 7:54 PM, Kyle Teague wrote: Thanks! I don't have access to a full fledged Hadoop cluster right now -- just trying to test out the software on a single machine. I changed the number of workers to 3 as I have one Task Tracker with a maximum of 4 map tasks and reduced the number of vertices to 500,000 and that fixed it. I changed the number of workers to 2, which On Wed, Sep 7, 2011 at 5:31 PM, Avery Ching wrote: Hi Kyle, Thanks for your question and welcome to Giraph! It looks like you couldn't get enough resources for the test to run on your hadoop instance. In this example, you are asking for 30 workers. You will need to be able to get 30 + 1 (master) = 31 map tasks to start the test. If Giraph can't get all 31 map tasks within a period of time, it will fail. Are you submitting this to an actual Hadoop cluster with at least 31 available map tasks? Avery On 9/7/11 2:13 PM, Kyle Teague wrote: I am trying to run the following command in pseudo-distributed mode from the Getting Started example page: hadoop jar giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 5000 -w 30 Here is the task log output: 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2011-09-07 15:41:34,529 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat: getOutputCommitter: Returning ImmutableOutputCommiter (does nothing). 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper: setup: jar file @ /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, using /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Made the directory _bsp/_defaultZkManagerDir/job_201109071501_0003 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Creating my filestamp _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home 0 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1 candidates when 1 required (polling period is 3000) on attempt 0 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager: createZooKeeperServerList: Creating the final ZooKeeper file '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home 0 ' 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: For task 0, got file 'zkServerList_new-host-3.home 0 ' (polling period is 3000) 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename 'zkServerList_new-host-3.home 0' 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Trying to delete old directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Creating file /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg in /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper with base port 22181 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Delete of zoo.cfg = false 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Attempting to start ZooKeeper server with command [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java, -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, org.apache.zookeeper.server.quorum.QuorumPeerMain, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg] in directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect to new-host-3.home:22181 with poll msecs = 3000 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Got ConnectException java.net.ConnectException: Connection refused at java.net.PlainSocke
Re: Can't get Getting Started example to work
Thanks! I don't have access to a full fledged Hadoop cluster right now -- just trying to test out the software on a single machine. I changed the number of workers to 3 as I have one Task Tracker with a maximum of 4 map tasks and reduced the number of vertices to 500,000 and that fixed it. I changed the number of workers to 2, which On Wed, Sep 7, 2011 at 5:31 PM, Avery Ching wrote: > Hi Kyle, > > Thanks for your question and welcome to Giraph! It looks like you couldn't > get enough resources for the test to run on your hadoop instance. In this > example, you are asking for 30 workers. You will need to be able to get 30 > + 1 (master) = 31 map tasks to start the test. If Giraph can't get all 31 > map tasks within a period of time, it will fail. Are you submitting this to > an actual Hadoop cluster with at least 31 available map tasks? > > Avery > > On 9/7/11 2:13 PM, Kyle Teague wrote: >> >> I am trying to run the following command in pseudo-distributed mode >> from the Getting Started example page: hadoop jar >> giraph-0.70-jar-with-dependencies.jar >> org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 5000 >> -w 30 >> >> Here is the task log output: >> >> 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader: >> Unable to load native-hadoop library for your platform... using >> builtin-java classes where applicable >> 2011-09-07 15:41:34,529 WARN >> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi >> already exists! >> 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat: >> getOutputCommitter: Returning ImmutableOutputCommiter (does nothing). >> 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper: >> setup: jar file @ >> >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, >> using >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar >> 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager: >> createCandidateStamp: Made the directory >> _bsp/_defaultZkManagerDir/job_201109071501_0003 >> 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager: >> createCandidateStamp: Creating my filestamp >> _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home >> 0 >> 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager: >> getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1 >> candidates when 1 required (polling period is 3000) on attempt 0 >> 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager: >> createZooKeeperServerList: Creating the final ZooKeeper file >> >> '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home >> 0 ' >> 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager: >> getZooKeeperServerList: For task 0, got file >> 'zkServerList_new-host-3.home 0 ' (polling period is 3000) >> 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager: >> getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename >> 'zkServerList_new-host-3.home 0' >> 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager: >> onlineZooKeeperServers: Trying to delete old directory >> >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper >> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: >> generateZooKeeperConfigFile: Creating file >> >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg >> in >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper >> with base port 22181 >> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: >> generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true >> 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: >> generateZooKeeperConfigFile: Delete of zoo.cfg = false >> 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager: >> onlineZooKeeperServers: Attempting to start ZooKeeper server with >> command >> [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java, >> -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC, >> -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp, >> >> /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, >> org.apache.zookeeper.server.quorum.QuorumPeerMain, >> >> /tmp/hadoop-kyle/
Re: Can't get Getting Started example to work
Hi Kyle, Thanks for your question and welcome to Giraph! It looks like you couldn't get enough resources for the test to run on your hadoop instance. In this example, you are asking for 30 workers. You will need to be able to get 30 + 1 (master) = 31 map tasks to start the test. If Giraph can't get all 31 map tasks within a period of time, it will fail. Are you submitting this to an actual Hadoop cluster with at least 31 available map tasks? Avery On 9/7/11 2:13 PM, Kyle Teague wrote: I am trying to run the following command in pseudo-distributed mode from the Getting Started example page: hadoop jar giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 5000 -w 30 Here is the task log output: 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2011-09-07 15:41:34,529 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat: getOutputCommitter: Returning ImmutableOutputCommiter (does nothing). 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper: setup: jar file @ /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, using /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Made the directory _bsp/_defaultZkManagerDir/job_201109071501_0003 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Creating my filestamp _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home 0 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1 candidates when 1 required (polling period is 3000) on attempt 0 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager: createZooKeeperServerList: Creating the final ZooKeeper file '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home 0 ' 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: For task 0, got file 'zkServerList_new-host-3.home 0 ' (polling period is 3000) 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename 'zkServerList_new-host-3.home 0' 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Trying to delete old directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Creating file /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg in /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper with base port 22181 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Delete of zoo.cfg = false 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Attempting to start ZooKeeper server with command [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java, -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, org.apache.zookeeper.server.quorum.QuorumPeerMain, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg] in directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect to new-host-3.home:22181 with poll msecs = 3000 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Got ConnectException java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432) at java.net.Socket.connect(Socket.java:529) at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:611) at o
Can't get Getting Started example to work
I am trying to run the following command in pseudo-distributed mode from the Getting Started example page: hadoop jar giraph-0.70-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -e 1 -s 3 -v -V 5000 -w 30 Here is the task log output: 2011-09-07 15:41:34,311 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2011-09-07 15:41:34,529 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists! 2011-09-07 15:41:34,641 WARN org.apache.giraph.bsp.BspOutputFormat: getOutputCommitter: Returning ImmutableOutputCommiter (does nothing). 2011-09-07 15:41:34,688 INFO org.apache.giraph.graph.GraphMapper: setup: jar file @ /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, using /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar 2011-09-07 15:41:34,694 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Made the directory _bsp/_defaultZkManagerDir/job_201109071501_0003 2011-09-07 15:41:34,695 INFO org.apache.giraph.zk.ZooKeeperManager: createCandidateStamp: Creating my filestamp _bsp/_defaultZkManagerDir/job_201109071501_0003/_task/new-host-3.home 0 2011-09-07 15:41:34,710 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Got [new-host-3.home] 1 hosts from 1 candidates when 1 required (polling period is 3000) on attempt 0 2011-09-07 15:41:34,711 INFO org.apache.giraph.zk.ZooKeeperManager: createZooKeeperServerList: Creating the final ZooKeeper file '_bsp/_defaultZkManagerDir/job_201109071501_0003/zkServerList_new-host-3.home 0 ' 2011-09-07 15:41:34,717 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: For task 0, got file 'zkServerList_new-host-3.home 0 ' (polling period is 3000) 2011-09-07 15:41:34,718 INFO org.apache.giraph.zk.ZooKeeperManager: getZooKeeperServerList: Found [new-host-3.home, 0] 2 hosts in filename 'zkServerList_new-host-3.home 0 ' 2011-09-07 15:41:34,720 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Trying to delete old directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Creating file /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg in /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper with base port 22181 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Make directory of _bspZooKeeper = true 2011-09-07 15:41:34,724 INFO org.apache.giraph.zk.ZooKeeperManager: generateZooKeeperConfigFile: Delete of zoo.cfg = false 2011-09-07 15:41:34,726 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Attempting to start ZooKeeper server with command [/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java, -Xmx256m, -XX:ParallelGCThreads=4, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=70, -XX:MaxGCPauseMillis=100, -cp, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/jars/job.jar, org.apache.zookeeper.server.quorum.QuorumPeerMain, /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper/zoo.cfg] in directory /tmp/hadoop-kyle/mapred/local/taskTracker/kyle/jobcache/job_201109071501_0003/work/_bspZooKeeper 2011-09-07 15:41:34,748 INFO org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Connect attempt 0 of 10 max trying to connect to new-host-3.home:22181 with poll msecs = 3000 2011-09-07 15:41:34,775 WARN org.apache.giraph.zk.ZooKeeperManager: onlineZooKeeperServers: Got ConnectException java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:432) at java.net.Socket.connect(Socket.java:529) at org.apache.giraph.zk.ZooKeeperManager.onlineZooKeeperServers(ZooKeeperManager.java:611) at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:419) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.secu
