Hello, The "clean" argument solved my current problem. However, I see more when I go on.
On "http://giraph.apache.org/build.html";, (1) There're commands for secure and non-secure Hadoop. What are the commands for YARN? (2) What do the compile, package, and test do respectively? Do I have to run each of them? Do I have to run them in order? When I ran "clean package", I saw the following error message. ===== 2013-07-16 11:07:42,383 FATAL [Thread-109] graph.GraphMapper (GraphMapper.java:uncaughtException(112)) - uncaughtException: OverrideExceptionHandler on thread Thread-109, msg = Failed to Start ResourceManager, exiting... org.apache.hadoop.yarn.YarnException: Failed to Start ResourceManager at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:78) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:567) at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper$2.run(MiniYARNCluster.java:164) Caused by: org.apache.hadoop.yarn.YarnException: java.net.BindException: Problem binding to [0.0.0.0:8033] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:139) at org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC.getServer(HadoopYarnProtoRPC.java:63) at org.apache.hadoop.yarn.ipc.YarnRPC.getServer(YarnRPC.java:52) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.start(AdminService.java:109) at org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68) ... 2 more Caused by: java.net.BindException: Problem binding to [0.0.0.0:8033] java.net.BindException: Address already in use; For more details see: http://wiki.apache.org/hadoop/BindException at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:716) at org.apache.hadoop.ipc.Server.bind(Server.java:370) at org.apache.hadoop.ipc.Server$Listener.<init>(Server.java:468) at org.apache.hadoop.ipc.Server.<init>(Server.java:1886) at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:982) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:376) at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:351) at org.apache.hadoop.ipc.RPC.getServer(RPC.java:705) at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.createServer(RpcServerFactoryPBImpl.java:169) at org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl.getServer(RpcServerFactoryPBImpl.java:132) ... 6 more ===== Does the "package" command start another YARN cluster? On "http://giraph.apache.org/pagerank.html";, (1) The last step, "Building and compiling page rank", is missing. (2) I successfully compiled the giraph-examples and have a working Hadoop YARN, how do I run the Page Rank program? Many thanks, Chui-hui On Mon, Jul 15, 2013 at 12:53 PM, Eli Reisman <[email protected]>wrote: > I think something else is wrong then, this same build works fine for me. I > think you need to do mvn targets "clean install" the first time, and > "compile" from then on? > > > On Mon, Jul 8, 2013 at 7:45 AM, Chui-Hui Chiu <[email protected]>wrote: > >> I run >> >> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn compile >> >> on the source code of the Giraph-1.1.0-SNAPSHOT, which is released on >> 05/21/2013, but still see the following message. >> >> ===== >> >> [INFO] Executed tasks >> [INFO] >> >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Building Apache Giraph Examples 1.1.0-SNAPSHOT >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Reactor Summary: >> [INFO] >> [INFO] Apache Giraph Parent .............................. SUCCESS >> [1.335s] >> [INFO] Apache Giraph Core ................................ SUCCESS >> [41.165s] >> [INFO] Apache Giraph Examples ............................ FAILURE >> [0.122s] >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] BUILD FAILURE >> [INFO] >> ------------------------------------------------------------------------ >> [INFO] Total time: 43.270s >> [INFO] Finished at: Mon Jul 08 09:37:16 CDT 2013 >> [INFO] Final Memory: 43M/724M >> [INFO] >> ------------------------------------------------------------------------ >> [ERROR] Failed to execute goal on project giraph-examples: Could not >> resolve dependencies for project >> org.apache.giraph:giraph-examples:jar:1.1.0-SNAPSHOT: Failure to find >> org.apache.giraph:giraph-core:jar:tests:1.1.0-SNAPSHOT in >> http://repo1.maven.org/maven2 was cached in the local repository, >> resolution will not be reattempted until the update interval of central has >> elapsed or updates are forced -> [Help 1] >> >> ===== >> >> When I apply the GIRAPH-688-1.patch, the patch utility reports that the >> patch has applied to the source code. I just compile it. >> >> >> On Thu, Jul 4, 2013 at 11:56 AM, Eli Reisman <[email protected]>wrote: >> >>> Yes, glad this was brought up, I'll post to the site wiki with my >>> explicit apologies for not doing it sooner. >>> >>> I posted the most recent "correct" way to build pure-YARN Giraph against >>> any hadoop version on GIRAPH-688 here: >>> >>> https://issues.apache.org/jira/browse/GIRAPH-688 >>> >>> The trick is, you have to put the -D mvn option first, and set the >>> "hadoop.version" Maven var. The versions as seen in "hadoop.version" are >>> -not- the same as the version profiles (-P option) that we build against. >>> So the correct format is: >>> >>> mvn -Dhadoop.version=2.0.3-alpha -Phadoop_yarn clean install >>> >>> if you want to just compile and not create our build products as in your >>> example, replace "clean install" with "compile". -P profile should be >>> "hadoop_yarn" for pure YARN builds, and "hadoop.version" appears as the >>> part of the hadoop version you see on the Apache Hadoop download site, >>> -after- the "hadoop-" part. So "hadoop-2.0.3-alpha" becomes for us >>> "-Dhadoop.version=2.0.3-alpha" >>> >>> To build Giraph so that it can run on a YARN-enabled Hadoop cluster that >>> -also- uses MapReduce v2 as the interface to the cluster (Giraph will still >>> think its running on MapReduce, not YARN) you might do this: >>> >>> mvn -Phadoop_2.0.3 clean install >>> >>> If you want to see how we label build profiles vs. "hadoop.version" in >>> Giraph builds, check our POM.xml files >>> >>> Thanks! Sorry for the confusion, this is a confusing thing since there >>> are 2 ways to "run Giraph on YARN" as detailed above. >>> >>> >>> >>> >>> On Wed, Jul 3, 2013 at 10:16 AM, Avery Ching <[email protected]> wrote: >>> >>>> Eli, any thoughts? >>>> >>>> >>>> On 7/3/13 9:27 AM, Chui-Hui Chiu wrote: >>>> >>>>> Hello, >>>>> >>>>> I tried to compile the Giraph-1.1.0-SNAPSHOT for hadoop_2.0.3 or >>>>> hadoop_yarn but all failed. >>>>> >>>>> The error message while the compile command is "mvn -Phadoop_yarn >>>>> compile" is >>>>> ===== >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] Reactor Summary: >>>>> [INFO] >>>>> [INFO] Apache Giraph Parent .............................. SUCCESS >>>>> [1.320s] >>>>> [INFO] Apache Giraph Core ..............................**.. FAILURE >>>>> [12.508s] >>>>> [INFO] Apache Giraph Examples ............................ SKIPPED >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] BUILD FAILURE >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] Total time: 14.473s >>>>> [INFO] Finished at: Wed Jul 03 11:05:44 CDT 2013 >>>>> [INFO] Final Memory: 14M/216M >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [ERROR] Failed to execute goal on project giraph-core: Could not >>>>> resolve dependencies for project >>>>> org.apache.giraph:giraph-core:**jar:1.1.0-SNAPSHOT: >>>>> The following artifacts could not be resolved: org.apache.hadoop:hadoop- >>>>> **common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION, >>>>> org.apache.hadoop:hadoop-**mapreduce-client-common:jar:** >>>>> SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION, >>>>> org.apache.hadoop:hadoop-**mapreduce-client-core:jar:SET_** >>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION, >>>>> org.apache.hadoop:hadoop-yarn-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION, >>>>> org.apache.hadoop:hadoop-yarn-**server-common:jar:SET_HADOOP_** >>>>> VERSION_USING_MVN_DASH_D_**OPTION, org.apache.hadoop:hadoop-yarn-** >>>>> server-resourcemanager:jar:**SET_HADOOP_VERSION_USING_MVN_**DASH_D_OPTION, >>>>> org.apache.hadoop:hadoop-yarn-**server-nodemanager:jar:SET_** >>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION, >>>>> org.apache.hadoop:hadoop-yarn-**server-tests:jar:tests:SET_** >>>>> HADOOP_VERSION_USING_MVN_DASH_**D_OPTION: Could not find artifact >>>>> org.apache.hadoop:hadoop-**common:jar:SET_HADOOP_VERSION_**USING_MVN_DASH_D_OPTION >>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1] >>>>> ===== >>>>> >>>>> >>>>> The error message while the compile command is "mvn -Phadoop_2.0.3 >>>>> compile" is >>>>> ===== >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] Reactor Summary: >>>>> [INFO] >>>>> [INFO] Apache Giraph Parent .............................. SUCCESS >>>>> [12.695s] >>>>> [INFO] Apache Giraph Core ..............................**.. SUCCESS >>>>> [2:10.916s] >>>>> [INFO] Apache Giraph Examples ............................ FAILURE >>>>> [2.286s] >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] BUILD FAILURE >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [INFO] Total time: 2:26.530s >>>>> [INFO] Finished at: Wed Jul 03 11:03:25 CDT 2013 >>>>> [INFO] Final Memory: 34M/348M >>>>> [INFO] ------------------------------**------------------------------* >>>>> *------------ >>>>> [ERROR] Failed to execute goal on project giraph-examples: Could not >>>>> resolve dependencies for project >>>>> org.apache.giraph:giraph-**examples:jar:1.1.0-SNAPSHOT: >>>>> Could not find artifact >>>>> org.apache.giraph:giraph-core:**jar:tests:1.1.0-SNAPSHOT >>>>> in central (http://repo1.maven.org/maven2**) -> [Help 1] >>>>> ===== >>>>> >>>>> Do I miss anything? >>>>> >>>>> >>>>> I also noticed that my Maven 3 download many files from the maven2 >>>>> folder on a remote server with the following prompt. >>>>> >>>>> Downloading: >>>>> http://repo1.maven.org/maven2/**org/apache/hadoop/.<http://repo1.maven.org/maven2/org/apache/hadoop/.> >>>>> .. >>>>> >>>>> Is this a problem? >>>>> >>>>> >>>>> Thanks, >>>>> Chui-hui >>>>> >>>> >>>> >>> >> >
