Hi Viraj, Your best bet is to take your modified hadoop tar ball (hadoop-0.20.10.tar.gz) and place it in .ant/cache/hadoop/core/sources/ along with the other Hadoop tar balls. Then run "ant -Dhadoop.version=0.20.10 package". Specifying hadoop.root manually is likely to cause problems. The build script will automatically grab hadoop-0.20.10.tar.gz from the ant cache, copy it to build/hadoopcore, unarchive the contents in this directory, and then set hadoop.root to point to build/hadoopcore/hadoop-0.20.10/
Carl On Mon, Jun 21, 2010 at 4:20 PM, Viraj Bhat <[email protected]> wrote: > Hi Carl, > > I worked on your suggestion and it seems to fail at another point. > > ant -Dhadoop.root=~/projects/newhadoop/hadoop-0.20.10/ > -Dhadoop.version=0.20.10 package -Doffline=true > > > > > > install-hadoopcore-internal: > > [untar] Expanding: > /homes/viraj/projects//hive-svn/trunk/build/hadoopcore/hadoop-0.17.2.1.tar.gz > into /homes/viraj/projects/hive-svn/trunk/build/hadoopcore > > > > > > BUILD FAILED > > /homes/viraj/projects/hive-svn/trunk/build.xml:160: The following error > occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/build.xml:103: The following error > occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/shims/build.xml:56: The following > error occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/build-common.xml:186: > /homes/viraj/projects/hive-svn/trunk/shims/~/projects/hadoop-0.20.10/bin not > found. > > Thanks Viraj > > > ------------------------------ > > *From:* Carl Steinbach [mailto:[email protected]] > *Sent:* Monday, June 21, 2010 9:14 AM > > *To:* [email protected] > *Subject:* Re: Compiling Hive against a different version of Hadoop 20 - > Attempt 2 > > > > Hi Viraj, > > > > Everything should work if you remove the "-Doffline=true" option. > > > > The problem you're running into is that the Hive build depends on > ivy-2.1.0.jar which gets downloaded to build/ivy/lib/ivy-2.1.0.jar as one of > the first steps in the build, but only if offline=false. This copy of ivy > gets blown away whenever you run 'ant clean', so you will need to either > leave the '-Doffline=true' switch out, or else run 'ant ivy-download' before > running 'ant package -Doffline=true'. > > > > Thanks. > > > > Carl > > On Sun, Jun 20, 2010 at 11:21 PM, Viraj Bhat <[email protected]> wrote: > > Hi all, > > I tried compiling this time by running ant clean first. > > > > Then I tried running > > ant -Dhadoop.root=~/projects/newhadoop/hadoop-0.20.10/ > -Dhadoop.version=0.20.10 package -Doffline=true > > > > BUILD FAILED > > /homes/viraj/projects/hive-svn/trunk/build.xml:160: The following error > occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/build.xml:103: The following error > occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/shims/build.xml:56: The following > error occurred while executing this line: > > /homes/viraj/projects/hive-svn/trunk/build-common.xml:127: You need Apache > Ivy 2.1 or later from http://ant.apache.org/ > > It could not be loaded from > http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar > > > > Viraj > ------------------------------ > > *From:* Viraj Bhat [mailto:[email protected]] > *Sent:* Tuesday, June 15, 2010 11:27 AM > *To:* [email protected]; [email protected] > *Subject:* RE: Compiling Hive against a different version of Hadoop 20 > > > > Hi Edward, > > Thanks for your help. I am compiling against a version of Hadoop which has > changed and removed some core classes. I call it Hadoop 20.9, but it is > different from the one available in the public hadoop repository. > > Viraj > > > ------------------------------ > > *From:* Edward Capriolo [mailto:[email protected]] > *Sent:* Tuesday, June 15, 2010 7:33 AM > *To:* [email protected] > *Subject:* Re: Compiling Hive against a different version of Hadoop 20 > > > > > > On Tue, Jun 15, 2010 at 1:24 AM, Aaron Kimball <[email protected]> wrote: > > Also try supplying this flag to ant: -Doffline=true > > > > On Mon, Jun 14, 2010 at 9:06 PM, Carl Steinbach <[email protected]> wrote: > > Hi Viraj, > > > > You can put your hadoop tarball in the cache directory that ivy uses > (probably ~/.ant/cache/hadoop/core/sources). This should prevent Ivy from > failing when it looks for version 0.20.9. > > > > Thanks. > > > > Carl > > > > On Mon, Jun 14, 2010 at 8:33 PM, Viraj Bhat <[email protected]> wrote: > > Hi all, > > I have a specific version of Hadoop 20, which I need to compile with Hive. > > I tried the following steps > > > > http://wiki.apache.org/hadoop/Hive/DeveloperGuide > > > > $ant -Dhadoop.root=~/myhome/hadoop-0.20.9 -Dhadoop.version=0.20.9 > > > > Is there a way to disable the ivy downloads. > > > > Even though the source tree for hadoop-0.20.9 is located on my home > directory, I get the following errors. > > > > ivy-resolve: > > [ivy:resolve] :: Ivy 2.1.0 - 20090925235825 :: http://ant.apache.org/ivy/:: > > [ivy:resolve] :: loading settings :: file = > /projects/hive-svn/trunk/ivy/ivysettings.xml > > [ivy:resolve] > > [ivy:resolve] :: problems summary :: > > [ivy:resolve] :::: WARNINGS > > [ivy:resolve] module not found: hadoop#core;0.20.9 > > [ivy:resolve] ==== hadoop-source: tried > > [ivy:resolve] -- artifact hadoop#core;0.20.9!hadoop.tar.gz(source): > > [ivy:resolve] > http://mirror.facebook.net/facebook/hive-deps/hadoop/core/hadoop-0.20.9/hadoop-0.20.9.tar.gz > > [ivy:resolve] ==== apache-snapshot: tried > > [ivy:resolve] > https://repository.apache.org/content/repositories/snapshots/hadoop/core/0.20.9/core-0.20.9.pom > > [ivy:resolve] -- artifact hadoop#core;0.20.9.!hadoop.tar.gz(source): > > [ivy:resolve] > https://repository.apache.org/content/repositories/snapshots/hadoop/core/0.20.9/hadoop-0.20.9.tar.gz > > [ivy:resolve] ==== maven2: tried > > [ivy:resolve] > http://repo1.maven.org/maven2/hadoop/core/0.20.9/core-0.20.9.pom > > [ivy:resolve] -- artifact hadoop#core;0.20.9.hadoop.tar.gz(source): > > [ivy:resolve] > http://repo1.maven.org/maven2/hadoop/core/0.20.9/core-0.20.9.tar.gz > > [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: > > [ivy:resolve] :: UNRESOLVED DEPENDENCIES :: > > [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: > > [ivy:resolve] :: hadoop#core;0.20.9: not found > > [ivy:resolve] :::::::::::::::::::::::::::::::::::::::::::::: > > > > [ivy:resolve] :: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS > > > > BUILD FAILED > > /projects/hive-svn/trunk/build.xml:160: The following error occurred while > executing this line: > > /projects/hive-svn/trunk/build.xml:103: The following error occurred while > executing this line: > > /projects/hive-svn/trunk/build-common.xml:151: impossible to resolve > dependencies: > > resolve failed - see output for details > > > > > > Thanks Viraj > > > > > > > Also, if I understand correctly, Hive should build the same result > regardless of 0.20.9 or 0.20.1 as everything in 0.20.X is API compatable. > That being true building with 0.20.1 works, so you may want to stick with > that. > > >
