Thanks Joydeep & Zheng for the feedback. We will apply the new 487 patch when it is ready.
Thanks! Eva. On 7/8/09 5:58 PM, "Joydeep Sen Sarma" <[email protected]> wrote: > Sorry - this is also needed as part of hive-487: > > In hadoop-20 - the -libjars has to come after the jar file/class > > Please try applying this patch to bin/ext/cli.sh > > --- cli.sh (revision 789726) > +++ cli.sh (working copy) > @@ -10,7 +10,7 @@ > exit 3; > fi > > - exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS > $HIVE_OPTS "$@" > + exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE > $HIVE_OPTS "$@" > } > > > I will add this to the 487 patch. > > -----Original Message----- > From: Zheng Shao [mailto:[email protected]] > Sent: Wednesday, July 08, 2009 5:51 PM > To: [email protected] > Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ... > > It seems hadoop 0.20 RunJar.java does not like the "-libjars" option. > Can you try remove the "-libjars xxx" from the command line? > It's added somewhere in bin/hive (or scripts that got called by bin/hive) > > Zheng > > On Wed, Jul 8, 2009 at 3:45 PM, Eva Tse<[email protected]> wrote: >> >> We set the env variable HIVE_AUX_JARS_PATH to a local path on the master >> node, which is on EC2 LS , which has a few jar files. When we start hive, it >> fails to start >> >> Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/ >> HIVE-487 patch >> >> It looks like the invocation command line is changed/different. Or maybe we >> have a setup issue? Any idea why this is happening? >> >> Thanks in advance, >> Eva. >> >> Here is the command line that starts hive (extracted from ps): >> /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20 >> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20 >> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console >> -Dhadoop.policy.file=hadoop-policy.xml -classpath >> /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/u >> sr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib >> /commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar: >> /usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons- >> httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/ >> lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/co >> mmons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0 >> .20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/h >> sqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/l >> ib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t- >> 0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/ >> jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoo >> p-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/h >> adoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/or >> o-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hado >> op-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3. >> jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1 >> /jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r >> 786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/Tes >> tSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.ja >> r:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r7866 >> 48/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/b >> uild/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/buil >> d/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/. >> ./lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/c >> ommons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derb >> y.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive- >> r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dis >> t/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_j >> dbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/l >> ib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/bu >> ild/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../ >> lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94 >> .jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib >> /hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r78 >> 6648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/d >> ist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8 >> .1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive >> -r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist >> /bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stri >> ngtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5 >> .jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.j >> ar:/home/dataeng/nfdev/dist/hiveudf.jar >> org.apache.hadoop.util.RunJar -libjars >> file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist >> /geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar >> /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar >> org.apache.hadoop.hive.cli.CliDriver -hiveconf >> hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home >> /dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar >> >> Error we get: >> [data...@master nfdev]$ hive >> Exception in thread "main" java.io.IOException: Error opening job jar: >> -libjars >> at org.apache.hadoop.util.RunJar.main(RunJar.java:90) >> Caused by: java.util.zip.ZipException: error in opening zip file >> at java.util.zip.ZipFile.open(Native Method) >> at java.util.zip.ZipFile.<init>(ZipFile.java:114) >> at java.util.jar.JarFile.<init>(JarFile.java:133) >> at java.util.jar.JarFile.<init>(JarFile.java:70) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:88) >> >> >> On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3 >> Hive) on a different cluster. Here is the last portion of the command line >> when starting hive and it would work: >> >> :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse >> /nfdev/dist/ParseNCCP.jar >> org.apache.hadoop.mapred.JobShell -libjars >> file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.j >> ar,file:///home/etse/nfdev/dist/ParseNCCP.jar >> /apps/hadoop/hive/bin/../lib/hive_cli.jar >> org.apache.hadoop.hive.cli.CliDriver -hiveconf >> hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/n >> fdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar > >
