Sorry - this is also needed as part of hive-487:
In hadoop-20 - the -libjars has to come after the jar file/class
Please try applying this patch to bin/ext/cli.sh
--- cli.sh (revision 789726)
+++ cli.sh (working copy)
@@ -10,7 +10,7 @@
exit 3;
fi
- exec $HADOOP jar $AUX_JARS_CMD_LINE ${HIVE_LIB}/hive_cli.jar $CLASS
$HIVE_OPTS "$@"
+ exec $HADOOP jar ${HIVE_LIB}/hive_cli.jar $CLASS $AUX_JARS_CMD_LINE
$HIVE_OPTS "$@"
}
I will add this to the 487 patch.
-----Original Message-----
From: Zheng Shao [mailto:[email protected]]
Sent: Wednesday, July 08, 2009 5:51 PM
To: [email protected]
Subject: Re: Can't start hive after setting HIVE_AUX_JARS_PATH ...
It seems hadoop 0.20 RunJar.java does not like the "-libjars" option.
Can you try remove the "-libjars xxx" from the command line?
It's added somewhere in bin/hive (or scripts that got called by bin/hive)
Zheng
On Wed, Jul 8, 2009 at 3:45 PM, Eva Tse<[email protected]> wrote:
>
> We set the env variable HIVE_AUX_JARS_PATH to a local path on the master
> node, which is on EC2 LS , which has a few jar files. When we start hive, it
> fails to start
>
> Hadoop/Hive version : Hadoop 0.20 unpatched and Hive trunk (r786648) w/
> HIVE-487 patch
>
> It looks like the invocation command line is changed/different. Or maybe we
> have a setup issue? Any idea why this is happening?
>
> Thanks in advance,
> Eva.
>
> Here is the command line that starts hive (extracted from ps):
> /usr/java/default/bin/java -Xmx1000m -Dhadoop.log.dir=/var/log/hadoop-0.20
> -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/lib/hadoop-0.20
> -Dhadoop.id.str= -Dhadoop.root.logger=INFO,console
> -Dhadoop.policy.file=hadoop-policy.xml -classpath
> /etc/hadoop-0.20/conf:/usr/java/default/lib/tools.jar:/usr/lib/hadoop-0.20:/usr/lib/hadoop-0.20/hadoop-0.20.0-1.cloudera-core.jar:/usr/lib/hadoop-0.20/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop-0.20/lib/commons-codec-1.3.jar:/usr/lib/hadoop-0.20/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop-0.20/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-logging-api-1.0.4.jar:/usr/lib/hadoop-0.20/lib/commons-net-1.4.1.jar:/usr/lib/hadoop-0.20/lib/core-3.1.1.jar:/usr/lib/hadoop-0.20/lib/hadoop-0.20.0-1.cloudera-fairscheduler.jar:/usr/lib/hadoop-0.20/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20/lib/jasper-compiler-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jasper-runtime-5.5.12.jar:/usr/lib/hadoop-0.20/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20/lib/jetty-6.1.14.jar:/usr/lib/hadoop-0.20/lib/jetty-util-6.1.14.jar:/usr/lib/hadoop-0.20/lib/junit-3.8.1.jar:/usr/lib/hadoop-0.20/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20/lib/log4j-1.2.15.jar:/usr/lib/hadoop-0.20/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop-0.20/lib/oro-2.0.8.jar:/usr/lib/hadoop-0.20/lib/servlet-api-2.5-6.1.14.jar:/usr/lib/hadoop-0.20/lib/slf4j-api-1.4.3.jar:/usr/lib/hadoop-0.20/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop-0.20/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-2.1.jar:/usr/lib/hadoop-0.20/lib/jsp-2.1/jsp-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../conf:/usr/lib/hive-r786648/build/dist/bin/../lib/TestSerDe.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/antlr-runtime-3.0.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/asm-3.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-collections-3.2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-lang-2.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/commons-logging-api-1.0.4.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/derby.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_common.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_exec.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_jdbc.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_metastore.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_serde.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/hive_service.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jdo2-api-2.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jline-0.9.94.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-core-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-enhancer-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/jpox-rdbms-1.2.2.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/json.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/junit-3.8.1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libfb303.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/libthrift.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/log4j-1.2.15.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/stringtemplate-3.1b1.jar:/usr/lib/hive-r786648/build/dist/bin/../lib/velocity-1.5.jar::/home/dataeng/nfdev/dist/ParseNCCP.jar:/home/dataeng/nfdev/dist/geoip.jar:/home/dataeng/nfdev/dist/hiveudf.jar
> org.apache.hadoop.util.RunJar -libjars
> file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
> /usr/lib/hive-r786648/build/dist/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/dataeng/nfdev/dist/ParseNCCP.jar,file:///home/dataeng/nfdev/dist/geoip.jar,file:///home/dataeng/nfdev/dist/hiveudf.jar
>
> Error we get:
> [data...@master nfdev]$ hive
> Exception in thread "main" java.io.IOException: Error opening job jar:
> -libjars
> at org.apache.hadoop.util.RunJar.main(RunJar.java:90)
> Caused by: java.util.zip.ZipException: error in opening zip file
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.<init>(ZipFile.java:114)
> at java.util.jar.JarFile.<init>(JarFile.java:133)
> at java.util.jar.JarFile.<init>(JarFile.java:70)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:88)
>
>
> On another version of Hadoop/Hive combination (i.e, 0.19 Hadoop and 0.3
> Hive) on a different cluster. Here is the last portion of the command line
> when starting hive and it would work:
>
> :/home/etse/nfdev/dist/geoip.jar:/home/etse/nfdev/dist/hiveudf.jar:/home/etse/nfdev/dist/ParseNCCP.jar
> org.apache.hadoop.mapred.JobShell -libjars
> file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
> /apps/hadoop/hive/bin/../lib/hive_cli.jar
> org.apache.hadoop.hive.cli.CliDriver -hiveconf
> hive.aux.jars.path=file:///home/etse/nfdev/dist/geoip.jar,file:///home/etse/nfdev/dist/hiveudf.jar,file:///home/etse/nfdev/dist/ParseNCCP.jar
--
Yours,
Zheng
