Re: Building Hive - cannot resolve dependencies

[Sasha Ovsankin]

Todd, Zheng -- The build process relies on downloading 4 different (!) hadoop versions from http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/ , a total of 160MB and fails if something goes wrong. This doesn't make a lot of sense -- why do you need 4 versions? Hive includes shims so that it can run against all of these versions. In truth it really only needs the jars from these releases, but since Hadoop isn't currently publishing its jars to a maven repo or anything, we have to get the tarballs. I agree that this is a little annoying, but once you've done it once it stays cached and you shouldn't have to worry about it. OK, I understand now. If all you need is jar files then the easiest would be to create a repository that contains these files. Since I am not very well versed with Ivy, I have created a Maven repository as a project on Google Code and put all 4 JAR files there. I guess with a bit of luck one can point Ivy to this repository instead of http://archive.apache.org and save the whole download mess. This repository contains only the JAR files and will download 20 times faster. The repo is located at: http://hadoop-via-maven.googlecode.com/svn/trunk/repo/org/apache/hadoop/hadoop-core/ I have tried to point Ivy to it by changing ivy/ivysettings.xml to :      <artifact pattern="http://hadoop-via-maven.googlecode.com/svn/trunk/repo/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]"/> , but this didn't work -- the build wants the complete tarballs. Guess I would need to do more digging into the build process :-( The project is on http://code.google.com/p/hadoop-via-maven/. I think there are quite a few folks craving for this kind of repository. Also, could you provide error message that explains how to manually download and install the dependencies, not only "try again". Or point to the doc that does. This is a good idea. Patches welcome :) This happens in build-common.xml, line 78 . Would love to help but I need to learn a bit more of Ant and Ivy in order to do this... BTW there might be a problem with http://archive.apache.org server -- it may be timing out connections which got me into an infinite loop of "try again". I haven't run into that before. This problem with 0.19 is new (didn't happen when we committed this built process). I've written up a sticky note on my monitor to take a look when I have a moment. For me, this consistently happened with all three 0.18 - 0.20. Most likely the tarball size and/or download time has something to do with this. I downloaded them with no problem to an EC2 machine though. Some bad guy sitting on the line... On 11/12/09 12:49 PM, Todd Lipcon wrote: On Thu, Nov 12, 2009 at 12:45 PM, Sasha Ovsankin <sa...@codebistro.com> wrote: Todd, Zheng -- The build process relies on downloading 4 different (!) hadoop versions from http://archive.apache.org/dist/hadoop/core/hadoop-0.20.0/ , a total of 160MB and fails if something goes wrong. This doesn't make a lot of sense -- why do you need 4 versions? Hive includes shims so that it can run against all of these versions. In truth it really only needs the jars from these releases, but since Hadoop isn't currently publishing its jars to a maven repo or anything, we have to get the tarballs. I agree that this is a little annoying, but once you've done it once it stays cached and you shouldn't have to worry about it.   Also, could you provide error message that explains how to manually download and install the dependencies, not only "try again". Or point to the doc that does. This is a good idea. Patches welcome :)   BTW there might be a problem with http://archive.apache.org server -- it may be timing out connections which got me into an infinite loop of "try again". I haven't run into that before. This problem with 0.19 is new (didn't happen when we committed this built process). I've written up a sticky note on my monitor to take a look when I have a moment. -Todd   Thanks, -- Sasha On Thu, Nov 12, 2009 at 2:30 AM, Zheng Shao <zsh...@gmail.com> wrote: Hi Todd, I don't have a clear idea but it seems the ivy download code has a bug? People get the files by doing "wget" on the same machine and it succeeds. If you have time, maybe we can debug into ivy's download code? Zheng On Thu, Nov 12, 2009 at 9:23 AM, Todd Lipcon <t...@cloudera.com> wrote: > Hey Zheng, > > What do we need to do to fix this? It seems to have bitten a number of > people by now. > > -Todd > > On Tue, Nov 10, 2009 at 3:50 PM, Zheng Shao <zsh...@gmail.com> wrote: >> >> I am forwarding an earlier email from the same mailing list by search >> for "Downloaded file size doesn't match expected Content Length": >> >> " >> Hi Rahul, >> >> Please follow these steps: >> >> 1) In your hive source directory run 'ant clean'. >> 2) remove the contents of ~/.ant/cache/hadoop/core/sources >> 3) Download the following files to ~/.ant/cache/hadoop/core/sources: >> >>      hadoop-0.17.2.1.tar.gz >>      hadoop-0.17.2.1.tar.gz.asc >>      hadoop-0.18.3.tar.gz >>      hadoop-0.18.3.tar.gz.asc >> >>      hadoop-0.19.0.tar.gz >>      hadoop-0.19.0.tar.gz.asc >>      hadoop-0.20.0.tar.gz >>      hadoop-0.20.0.tar.gz.asc >> >> 4) For each hadoop-xxx.tar.gz file, compute the sha1 checksum using >> sha1sum, and verify that it matches the sha1 checksum in the >> corresponding .asc file. >> >> If it does not match then the file is corrupt and you need to try >> downloading it again. >> >> 5) Try building Hive again following the instructions on the wiki. You >> shouldn't have any problems if you verified the checksums. >> >> As an additional note, if you don't care about support for Hadoop >> 0.17.2.1, or 0.18, etc, you can disable support for these versions >> (and skip the download) by removing the references to these versions >> that shims/build.xml and shims/ivy.xml >> >> >> >>    Also I want to use hive on top of current working hadoop cluster. >>    Please provide some inputs. >> >> >> You need to set HADOOP_HOME and make sure that HADOOP_HOME/conf (or >> HADOOP_CONF_DIR if you have this variable set) contains valid >> configuration files for your current working hadoop cluster. See the >> following page for more information: >> http://wiki.apache.org/hadoop/GettingStartedWithHadoop >> >> Thanks. >> >> Carl >> " >> >> On Tue, Nov 10, 2009 at 3:39 PM, Sasha Ovsankin <sa...@codebistro.com> >> wrote: >> > Hello, all -- >> > >> > I am trying to build as per GettingStarted and am getting this error >> > from >> > Ivy: >> > >> > Downloaded file size doesn't match expected Content Length for >> > ...hadoop-0.19.0.tar.gz. Please retry. >> > >> > This error repeats 4 times for 4 different versions of Hadoop and the >> > build >> > stops. >> > >> > I have tried on Mac and Windows with the same results. How can I build >> > Hive? >> > >> > Thanks for your help, >> > -- Sasha >> > >> > P.S. This is try#3 to get over the spam filter. >> > >> > >> >> >> >> -- >> Yours, >> Zheng > > -- Yours, Zheng