Thanks for pointing out Ravi. We are working on fixing it. The jars that you mention are created as part of MR-279 build.
For building MR-279, all you have to do in mvn clean install -DskipTests and that should work (given you have already installed common and hdfs from trunk). thanks mahadev On Fri, Aug 5, 2011 at 3:59 AM, Ravi Teja <[email protected]> wrote: > Hi, > > The Hudson build is failing now with the following error. > I am also facing following error. How is this resolved, as this is the > first > step of the build by which, even the yarn jars don't compile, neither the > jars are present in the repositories. > > Even if we install the jars manually from the previous Hudson > build,it may fail the compilation if any code changes are done after it. > > From where are the jars taken, to resolve this dependency? > > Please help me resolving this issue. > > > [ivy:resolve] > :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:resolve] :: UNRESOLVED > DEPENDENCIES :: > [ivy:resolve] > :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:resolve] :: > org.apache.hadoop#yarn-server-common;0.279.0: not found > [ivy:resolve] :: > org.apache.hadoop#hadoop-mapreduce-client-core;0.279.0: not found > [ivy:resolve] :: > org.apache.hadoop#yarn-common;0.279.0: not found > [ivy:resolve] > :::::::::::::::::::::::::::::::::::::::::::::: > [ivy:resolve] > [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS > > > Thanks and Regards, > Ravi Teja > > > -----Original Message----- > From: Ravi Teja [mailto:[email protected]] > Sent: Thursday, July 28, 2011 5:09 PM > To: '[email protected]' > Subject: Re: trouble building MR-279 branch > > > Hi, > > I have followed the procedure mentioned below, for building MRv2, but was > not able to succeed with the task. I have got some compilation errors. > > > Please help me resolving this issue. > > > > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Build Order: > [INFO] > [INFO] yarn-api > [INFO] yarn-common > [INFO] yarn-server-common > [INFO] yarn-server-nodemanager > [INFO] yarn-server-resourcemanager > [INFO] yarn-server-tests > [INFO] yarn-server > [INFO] yarn > [INFO] hadoop-mapreduce-client-core > [INFO] hadoop-mapreduce-client-common > [INFO] hadoop-mapreduce-client-shuffle > [INFO] hadoop-mapreduce-client-app > [INFO] hadoop-mapreduce-client-hs > [INFO] hadoop-mapreduce-client-jobclient > [INFO] hadoop-mapreduce-client > [INFO] hadoop-mapreduce > [INFO] > > [INFO] > ------------------------------------------------------------------------ > [INFO] Building yarn-api 1.0-SNAPSHOT > [INFO] > ------------------------------------------------------------------------ > [INFO] > [INFO] --- maven-antrun-plugin:1.6:run > (create-protobuf-generated-sources-directory) @ yarn-api --- > [INFO] Executing tasks > > main: > [INFO] Executed tasks > [INFO] > [INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ yarn-api --- > [INFO] > [INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @ yarn-api > --- > [INFO] Source directory: > D:\Code\MR-279\mapreduce\yarn\yarn-api\target\generated-sources\proto > added. > > [INFO] > [INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ > yarn-api --- > [INFO] Using 'UTF-8' encoding to copy filtered resources. > [INFO] skip non existing resourceDirectory > D:\Code\MR-279\mapreduce\yarn\yarn-api\src\main\resources > [INFO] > [INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ yarn-api > --- > [INFO] Compiling 121 source files to > D:\Code\MR-279\mapreduce\yarn\yarn-api\target\classes > [INFO] ------------------------------------------------------------- > [ERROR] COMPILATION ERROR : > [INFO] ------------------------------------------------------------- > [ERROR] > > \Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap > i\records\impl\pb\ContainerPBImpl.java:[22,33] package > org.apache.commons.logging does not exist > [ERROR] > > \Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap > i\records\impl\pb\ContainerPBImpl.java:[23,33] package > org.apache.commons.logging does not exist > [INFO] 2 errors > [INFO] ------------------------------------------------------------- > [INFO] > ------------------------------------------------------------------------ > [INFO] Reactor Summary: > [INFO] > [INFO] yarn-api .......................................... FAILURE [5.610s] > [INFO] yarn-common ....................................... SKIPPED > [INFO] yarn-server-common ................................ SKIPPED > [INFO] yarn-server-nodemanager ........................... SKIPPED > [INFO] yarn-server-resourcemanager ....................... SKIPPED > [INFO] yarn-server-tests ................................. SKIPPED > [INFO] yarn-server ....................................... SKIPPED > [INFO] yarn .............................................. SKIPPED > [INFO] hadoop-mapreduce-client-core ...................... SKIPPED > [INFO] hadoop-mapreduce-client-common .................... SKIPPED > [INFO] hadoop-mapreduce-client-shuffle ................... SKIPPED > [INFO] hadoop-mapreduce-client-app ....................... SKIPPED > [INFO] hadoop-mapreduce-client-hs ........................ SKIPPED > [INFO] hadoop-mapreduce-client-jobclient ................. SKIPPED > [INFO] hadoop-mapreduce-client ........................... SKIPPED > [INFO] hadoop-mapreduce .................................. SKIPPED > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 5.844s > [INFO] Finished at: Thu Jul 28 16:03:07 IST 2011 > [INFO] Final Memory: 9M/35M > [INFO] > ------------------------------------------------------------------------ > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:2.3.2:compile > (default-compile) on project yarn-api: Compilation failure: Compilation > failure: > [ERROR] > > \Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap > i\records\impl\pb\ContainerPBImpl.java:[22,33] package > org.apache.commons.logging does not exist > [ERROR] > > \Code\MR-279\mapreduce\yarn\yarn-api\src\main\java\org\apache\hadoop\yarn\ap > i\records\impl\pb\ContainerPBImpl.java:[23,33] package > org.apache.commons.logging does not exist > [ERROR] -> [Help 1] > > Regards, > Ravi Teja > > > > > > > -----Original Message----- > From: Josh Wills [mailto:[email protected]] > Sent: Saturday, July 02, 2011 6:34 AM > To: [email protected] > Subject: Re: trouble building MR-279 branch > > Hey Anupam, > > I think Jim is asking about building non-MR application masters using the > RM/NM. I think Arun has promised to post a design doc soon. In the > meantime, > I'm just reading through the code and fixing little issues I come across > (e.g., https://issues.apache.org/jira/browse/MAPREDUCE-2630 and > https://issues.apache.org/jira/browse/MAPREDUCE-2633). > > Jim, what sort of app are you looking to build? I'd be happy to have some > company wading through the code. > > Best, > Josh > > On Fri, Jul 1, 2011 at 3:02 PM, Anupam Seth <[email protected]> wrote: > > > Hi Jim, > > > > Not sure if this helps at all, but not sure if you could get an idea by > > looking at the included examples: > > > > > > > > http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/exa > mples/org/apache/hadoop/examples > > > > that can be run as: > > > > hadoop jar $HADOOP_MAPRED_HOME/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar > > wordcount input output > > > > Regards, > > Anupam > > > > -----Original Message----- > > From: Jim Falgout [mailto:[email protected]] > > Sent: Friday, July 01, 2011 4:23 PM > > To: [email protected] > > Subject: RE: trouble building MR-279 branch > > > > Thanks Anupam! I had skipped the step to get protobuf into the maven > > repository. Also, I had Apache Forrest version 0.9 installed. That caused > > the builds to fail on generating some docs. Installed version 0.8 and it > > worked smoothly. > > > > On another note: do you know of any design docs for building an > application > > executor using the new resource manager? Just wondering how to get > started? > > > > -----Original Message----- > > From: Anupam Seth [mailto:[email protected]] > > Sent: Friday, July 01, 2011 2:10 PM > > To: [email protected] > > Subject: RE: trouble building MR-279 branch > > > > Hi Jim, > > > > Following are some notes (thanks to Jeff Naisbitt & Bobby Evans). > > > > Build YARN > > ========== > > These are in addition to what you would normally do. > > > > Install protobuf > > use Protobuf 2.4.0a (Download from > > http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 > > and it worked > > > > install the protoc executable (configure, make, make install) > > install the maven artifact (cd java; mvn install) > > > > Installing protoc requires gcc 4.1.x or higher (RHEL5). > > > > If the make step fails with (Valid until a fix is released for protobuf > > 2.4.0a) ./google/protobuf/descriptor.h:1152: error: > > > > `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' > > is private Replace descriptor.cc with > > > > http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/de > scriptor.cc > > > > Setup environment > > > > export MAVEN_OPTS=-Xmx512m > > export LD_LIBRARY_PATH=/usr/local/lib > > > > Checkout the MR-279 branch > > svn checkout > http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279 > > > > Build common > > > > cd MR-279/common > > ant veryclean mvn-install tar > > > > Build hdfs > > > > cd ../hdfs > > ant veryclean mvn-install -Dresolvers=internal tar > > > > Build yarn > > > > cd ../mapreduce > > mvn clean install assembly:assembly > > # If tests are failing, add this option to skip them: > > -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must > > also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean > > install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>) > > ant veryclean jar jar-test tar -Dresolvers=internal > > > > -----Original Message----- > > From: Jim Falgout [mailto:[email protected]] > > Sent: Friday, July 01, 2011 2:00 PM > > To: [email protected] > > Subject: trouble building MR-279 branch > > > > Hi, > > > > I've checked out the MR-279 branch and am having trouble getting it to > > build. Any docs/blogs/notes around that may give some hints? > > > > The build is unable to resolve the yarn-server even though I ran a > > "mvn-install" in the yarn source area to install it in the local > repository. > > > > Thanks, > > Jim. > > > > > > > >
