Yeah, so just to be clear under pig > 0.10 the issue seems to be exactly as you describe + issue occurs whenever you specify in the -Dpig.additional.jars a directory path instead of the file path. This is quite often happening because its advised on forums to include HIVE_HOME and HADOOP_HOME in the PIG_CLASSPATH which is then passed to -Dpig.additional.jars. I put a comment in the jira ticket.
Thanks again Cheolsoo! On 13 November 2012 17:18, Cheolsoo Park <[email protected]> wrote: > Hi Michal, > > Thanks for sharing your workaround. > > I think that Pig should be able to handle empty file names in > -Dpig.additional.jars, so users don't have to spend hours to debug problems > like this. So I filed a JIRA: > https://issues.apache.org/jira/browse/PIG-3046 > > We will get this fixed in a future release. > > Thanks, > Cheolsoo > > On Tue, Nov 13, 2012 at 7:40 AM, Michał Czerwiński < > [email protected] > > wrote: > > > Oh well I > > changed > > > PIG_CLASSPATH="$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf" > > into > > PIG_CLASSPATH="$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar" > > > > having still hive libraries loaded via > > for file in $HIVE_HOME/lib/*.jar; do > > #echo "==> Adding $file" > > PIG_CLASSPATH="$PIG_CLASSPATH:$file" > > done > > > > and that seems to be working fine now, thanks a lot for help debugging > it! > > > > On 13 November 2012 15:16, Michał Czerwiński <[email protected] > > >wrote: > > > > > Right, it looks like that: > > > > > > 2012-11-13 15:13:57,100 [main] DEBUG > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > > - Adding jar to DistributedCache: > > > file:/opt/hcat/share/hcatalog/hcatalog-0.4.0.jar > > > 2012-11-13 15:13:57,428 [main] DEBUG > > > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > > - Adding jar to DistributedCache: file:/usr/lib/hive/conf/ > > > 2012-11-13 15:13:57,433 [main] ERROR org.apache.pig.tools.grunt.Grunt - > > > ERROR 2017: Internal error creating job configuration. > > > Details at logfile: /opt/pig/trunk/pig_1352819617642.log > > > > > > #> ls -la /usr/lib/hive/conf/ > > > total 88 > > > drwxr-xr-x 2 root root 4096 2012-11-12 17:48 . > > > drwxr-xr-x 8 root root 4096 2012-11-09 17:29 .. > > > -rw-r--r-- 1 root root 39451 2012-11-08 10:24 hive-default.xml > > > -rw-r--r-- 1 root root 1408 2012-11-08 11:22 hive-env.sh > > > -rw-r--r-- 1 root root 1410 2012-11-08 10:24 hive-env.sh.template > > > -rw-r--r-- 1 root root 1637 2012-11-08 10:24 > hive-exec-log4j.properties > > > -rw-r--r-- 1 root root 2005 2012-11-08 10:24 hive-log4j.properties > > > -rw-r--r-- 1 root root 4055 2012-11-08 11:22 hive-site-client.xml.tpl > > > -rw-rw-r-- 1 root root 4879 2012-11-09 15:30 hive-site.xml > > > -rw-r--r-- 1 root root 4903 2012-11-09 15:30 hive-site.xml.PIG.tpl > > > -rw-r--r-- 1 root root 3634 2012-11-08 11:22 hive-site.xml.tpl > > > > > > On 12 November 2012 18:45, Cheolsoo Park <[email protected]> > wrote: > > > > > >> Can you try to print out debug message by adding "-d DEBUG" to the Pig > > >> command? It will print which additional files are added to distributed > > >> cache as follows: > > >> > > >> 2012-11-12 10:41:58,908 [main] DEBUG > > >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > >> - Adding jar to DistributedCache: > > >> file:/home/cheolsoo/apache-ant-1.8.4/lib/ant-antlr.jar > > >> 2012-11-12 10:41:59,099 [main] DEBUG > > >> > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > >> - Adding jar to DistributedCache: file:/etc/hadoop-0.20/conf.pseudo/ > > >> > > >> This will tell you which file it was shipping right before failed. > That > > >> will probably give you a hint on where to look into further. > > >> > > >> Thanks, > > >> Cheolsoo > > >> > > >> > > >> On Mon, Nov 12, 2012 at 10:29 AM, Michał Czerwiński < > > >> [email protected]> wrote: > > >> > > >> > Seems like exactly the same error. > > >> > > > >> > I do it like that: > > >> > > > >> > > export JAVA_HOME=$(readlink -f /usr/bin/java | sed "s:bin/java::") > > >> > which resolves to /usr/lib/jvm/java-6-sun-1.6.0.21/jre/ > > >> > > > >> > > bin/pig > > >> > > > >> > > > >> > > > -Dpig.additional.jars=/opt/hcat/share/hcatalog/hcatalog-0.4.0.jar:/usr/lib/hive/conf:/usr/lib/hadoop-0.20/conf:/usr/lib/hive/lib/ant-contrib-1.0b3.jar:/usr/lib/hive/lib/antlr-runtime-3.0.1.jar:/usr/lib/hive/lib/asm-3.1.jar:/usr/lib/hive/lib/avro-1.5.4.jar:/usr/lib/hive/lib/avro-ipc-1.5.4.jar:/usr/lib/hive/lib/avro-mapred-1.5.4.jar:/usr/lib/hive/lib/commons-cli-1.2.jar:/usr/lib/hive/lib/commons-codec-1.3.jar:/usr/lib/hive/lib/commons-collections-3.2.1.jar:/usr/lib/hive/lib/commons-dbcp-1.4.jar:/usr/lib/hive/lib/commons-lang-2.4.jar:/usr/lib/hive/lib/commons-logging-1.0.4.jar:/usr/lib/hive/lib/commons-logging-api-1.0.4.jar:/usr/lib/hive/lib/commons-pool-1.5.4.jar:/usr/lib/hive/lib/datanucleus-connectionpool-2.0.3.jar:/usr/lib/hive/lib/datanucleus-core-2.0.3-ZD5977-CDH5293.jar:/usr/lib/hive/lib/datanucleus-enhancer-2.0.3.jar:/usr/lib/hive/lib/datanucleus-rdbms-2.0.3.jar:/usr/lib/hive/lib/derby.jar:/usr/lib/hive/lib/guava-r06.jar:/usr/lib/hive/lib/haivvreo-1.0.7-cdh-2.jar:/usr/lib/hive/lib/high-scale-lib-1.1.1.jar:/usr/lib/hive/lib/hive-anttasks-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-cli-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-common-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-contrib-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-exec-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-hbase-handler-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-jdbc-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-metastore-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-serde-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-service-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/hive-shims-0.7.1-cdh3u5.jar:/usr/lib/hive/lib/jackson-core-asl-1.7.3.jar:/usr/lib/hive/lib/jackson-jaxrs-1.7.3.jar:/usr/lib/hive/lib/jackson-mapper-asl-1.7.3.jar:/usr/lib/hive/lib/jackson-xc-1.7.3.jar:/usr/lib/hive/lib/jdo2-api-2.3-ec.jar:/usr/lib/hive/lib/jline-0.9.94.jar:/usr/lib/hive/lib/json.jar:/usr/lib/hive/lib/junit-3.8.1.jar:/usr/lib/hive/lib/libfb303.jar:/usr/lib/hive/lib/libthrift.jar:/usr/lib/hive/lib/log4j-1.2.15.jar:/usr/lib/hive/lib/slf4j-api-1.6.1.jar:/usr/lib/hive/lib/slf4j-log4j12-1.6.1.jar:/usr/lib/hive/lib/snappy-java-1.0.3.2.jar:/usr/lib/hive/lib/stringtemplate-3.1b1.jar:/usr/lib/hive/lib/thrift-0.5.0.jar:/usr/lib/hive/lib/thrift-fb303-0.5.0.jar:/usr/lib/hive/lib/velocity-1.5.jar > > >> > > > >> > which is basically > > >> > echo bin/pig -Dpig.additional.jars="$PIG_CLASSPATH" "$@" > > >> > > > >> > grunt> A = load 'xxx.yyy' using org.apache.hcatalog.pig.HCatLoader; > > >> > 2012-11-12 18:22:38,398 [main] INFO hive.metastore - Trying to > > connect > > >> to > > >> > metastore with URI thrift://hcatalog:10002 > > >> > 2012-11-12 18:22:38,506 [main] INFO hive.metastore - Connected to > > >> > metastore. > > >> > grunt> B = FILTER A BY keyword=='FU'; > > >> > grunt> ll = LIMIT B 10; > > >> > grunt> dump ll; > > >> > > > >> > 012-11-12 18:22:47,567 [main] INFO > > >> > org.apache.pig.tools.pigstats.ScriptState - Pig features used in > the > > >> > script: FILTER,LIMIT > > >> > 2012-11-12 18:22:47,901 [main] INFO > > >> > > > >> > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler > - > > >> > File concatenation threshold: 100 optimistic? false > > >> > 2012-11-12 18:22:48,026 [main] INFO > > >> > > > >> > > > >> > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > >> > - MR plan size before optimization: 2 > > >> > 2012-11-12 18:22:48,026 [main] INFO > > >> > > > >> > > > >> > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer > > >> > - MR plan size after optimization: 2 > > >> > 2012-11-12 18:22:48,249 [main] INFO > > >> > org.apache.pig.tools.pigstats.ScriptState - Pig script settings are > > >> added > > >> > to the job > > >> > 2012-11-12 18:22:48,333 [main] INFO > > >> > > > >> > > > >> > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > >> > - mapred.job.reduce.markreset.buffer.percent is not set, set to > > default > > >> 0.3 > > >> > 2012-11-12 18:22:48,629 [main] INFO hive.metastore - Trying to > > connect > > >> to > > >> > metastore with URI thrift://hcatalog:10002 > > >> > 2012-11-12 18:22:48,630 [main] INFO hive.metastore - Connected to > > >> > metastore. > > >> > 2012-11-12 18:22:49,476 [main] INFO > > >> > > > >> > > > >> > > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler > > >> > - Setting Parallelism to 1 > > >> > 2012-11-12 18:22:49,705 [main] ERROR > org.apache.pig.tools.grunt.Grunt > > - > > >> > ERROR 2017: Internal error creating job configuration. > > >> > Details at logfile: /opt/pig/trunk/pig_1352744549197.log > > >> > > > >> > The content of the above mentioned file is: > > >> > > > >> > Pig Stack Trace > > >> > --------------- > > >> > ERROR 2017: Internal error creating job configuration. > > >> > > > >> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: > Unable > > >> to > > >> > open iterator for alias ll > > >> > at org.apache.pig.PigServer.openIterator(PigServer.java:841) > > >> > at > > >> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) > > >> > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > > >> > at org.apache.pig.Main.run(Main.java:535) > > >> > at org.apache.pig.Main.main(Main.java:154) > > >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > >> > at > > >> > > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >> > at java.lang.reflect.Method.invoke(Method.java:597) > > >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > > >> > Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store > > >> alias > > >> > ll > > >> > at org.apache.pig.PigServer.storeEx(PigServer.java:940) > > >> > at org.apache.pig.PigServer.store(PigServer.java:903) > > >> > at org.apache.pig.PigServer.openIterator(PigServer.java:816) > > >> > ... 12 more > > >> > Caused by: > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: > > >> > ERROR 2017: Internal error creating job configuration. > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177) > > >> > at org.apache.pig.PigServer.launchPlan(PigServer.java:1269) > > >> > at > > >> > > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254) > > >> > at org.apache.pig.PigServer.storeEx(PigServer.java:936) > > >> > ... 14 more > > >> > Caused by: java.lang.IllegalArgumentException: Can not create a Path > > >> from > > >> > an empty string > > >> > at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) > > >> > at org.apache.hadoop.fs.Path.<init>(Path.java:90) > > >> > at org.apache.hadoop.fs.Path.<init>(Path.java:45) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.shipToHDFS(JobControlCompiler.java:1455) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.putJarOnClassPathThroughDistributedCache(JobControlCompiler.java:1432) > > >> > at > > >> > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:508) > > >> > ... 19 more > > >> > > > >> > > > >> > > > ================================================================================ > > >> > > > >> > Thanks for help! > > >> > > > >> > On 12 November 2012 18:09, Cheolsoo Park <[email protected]> > > wrote: > > >> > > > >> > > Interesting because I am able to reproduce your error by passing > > >> > > <jar1>::<jar2> to -Dpig.additional.jar. Setting PIG_CLASSPATH > itself > > >> with > > >> > > :: seems fine, but passing it to -Dpig.additional.jar is not. > > >> > > > > >> > > If you still run into a failure, can you double-check whether > you're > > >> > seeing > > >> > > the same error or a different one? You shouldn't see the same > error. > > >> > > > > >> > > Cheolsoo > > >> > > > > >> > > On Mon, Nov 12, 2012 at 9:59 AM, Michał Czerwiński < > > >> > > [email protected] > > >> > > > wrote: > > >> > > > > >> > > > Nice spot on, but it seems that's not the problem...I am able to > > run > > >> > > > this successfully on pig 0.9 and 0.10 any ideas how further I > can > > >> debug > > >> > > the > > >> > > > problem? > > >> > > > > > >> > > > I also double checked all the variables, seems to be fine: > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > > PIG_CLASSPATH=/opt/hcat/share/hcatalog/hcatalog-0.4.0.jar:/usr/lib/hive/conf:/usr/lib/hadoop-0.20/conf > > >> > > > > > >> > > > /usr/lib/hive/conf and /usr/lib/hadoop-0.20/conf also exist. > > >> > > > > > >> > > > Thanks. > > >> > > > > > >> > > > On 12 November 2012 17:37, Cheolsoo Park <[email protected] > > > > >> > wrote: > > >> > > > > > >> > > > > Hi Michal, > > >> > > > > > > >> > > > > Caused by: java.lang.IllegalArgumentException: Can not create > a > > >> Path > > >> > > > > from an empty string > > >> > > > > at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) > > >> > > > > at org.apache.hadoop.fs.Path.<init>(Path.java:90) > > >> > > > > at org.apache.hadoop.fs.Path.<init>(Path.java:45 > > >> > > > > > > >> > > > > Your error message indicates that there is a typo somewhere in > > >> > paths. I > > >> > > > > believe that your PIG_CLASSPATH is the problem: > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar::$HIVE_HOME/conf:$HADOOP_HOME/conf > > >> > > > > > > >> > > > > You have a double colon :: in the middle, and that will be > > >> > interpreted > > >> > > as > > >> > > > > an empty string. > > >> > > > > > > >> > > > > Thanks, > > >> > > > > Cheolsoo > > >> > > > > > > >> > > > > On Mon, Nov 12, 2012 at 8:47 AM, Michał Czerwiński < > > >> > > > > [email protected] > > >> > > > > > wrote: > > >> > > > > > > >> > > > > > I am trying to use pig 0.11 and pig trunk (currently 0.12) > > >> because > > >> > > pig > > >> > > > > 0.10 > > >> > > > > > seems to be having issues with python udf... > > >> > > > > > > > >> > > > > > According to this > > >> > > > > > > http://www.mail-archive.com/[email protected]/msg05837.html > > >> > > > > > > > >> > > > > > " after replacing pig.jar and pig-withouthadoop.jar with the > > >> > > > > > 0.11 ones from the svn trunk, they work like a charm." > > >> > > > > > > > >> > > > > > Well this is clearly not the case for me... > > >> > > > > > > > >> > > > > > The error I get is: > > >> > > > > > > > >> > > > > > Pig Stack Trace > > >> > > > > > --------------- > > >> > > > > > ERROR 2017: Internal error creating job configuration. > > >> > > > > > > > >> > > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR > > 1066: > > >> > > Unable > > >> > > > to > > >> > > > > > open iterator for alias ll > > >> > > > > > at > org.apache.pig.PigServer.openIterator(PigServer.java:841) > > >> > > > > > at > > >> > > > > > > >> > > > > >> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:696) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:320) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170) > > >> > > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > > >> > > > > > at org.apache.pig.Main.run(Main.java:535) > > >> > > > > > at org.apache.pig.Main.main(Main.java:154) > > >> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > Method) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > >> > > > > > at java.lang.reflect.Method.invoke(Method.java:597) > > >> > > > > > at org.apache.hadoop.util.RunJar.main(RunJar.java:197) > > >> > > > > > Caused by: org.apache.pig.PigException: ERROR 1002: Unable > to > > >> store > > >> > > > alias > > >> > > > > > ll > > >> > > > > > at org.apache.pig.PigServer.storeEx(PigServer.java:940) > > >> > > > > > at org.apache.pig.PigServer.store(PigServer.java:903) > > >> > > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:816) > > >> > > > > > ... 12 more > > >> > > > > > Caused by: > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobCreationException: > > >> > > > > > ERROR 2017: Internal error creating job configuration. > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:848) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:294) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177) > > >> > > > > > at org.apache.pig.PigServer.launchPlan(PigServer.java:1269) > > >> > > > > > at > > >> > > > > > > > >> > > > > > >> > > > org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1254) > > >> > > > > > at org.apache.pig.PigServer.storeEx(PigServer.java:936) > > >> > > > > > ... 14 more > > >> > > > > > Caused by: java.lang.IllegalArgumentException: Can not > create > > a > > >> > Path > > >> > > > from > > >> > > > > > an empty string > > >> > > > > > at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82) > > >> > > > > > at org.apache.hadoop.fs.Path.<init>(Path.java:90) > > >> > > > > > at org.apache.hadoop.fs.Path.<init>(Path.java:45) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.shipToHDFS(JobControlCompiler.java:1455) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.putJarOnClassPathThroughDistributedCache(JobControlCompiler.java:1432) > > >> > > > > > at > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.getJob(JobControlCompiler.java:508) > > >> > > > > > ... 19 more > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > ================================================================================ > > >> > > > > > > > >> > > > > > I am using the following startup script: > > >> > > > > > > > >> > > > > > export HADOOP_HOME=/usr/lib/hadoop-0.20 > > >> > > > > > export HCAT_HOME=/opt/hcat > > >> > > > > > export HIVE_HOME=/usr/lib/hive > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar::$HIVE_HOME/conf:$HADOOP_HOME/conf > > >> > > > > > > > >> > > > > > for file in $HIVE_HOME/lib/*.jar; do > > >> > > > > > echo "==> Adding $file" > > >> > > > > > PIG_CLASSPATH=$PIG_CLASSPATH:$file > > >> > > > > > done > > >> > > > > > > > >> > > > > > export PIG_OPTS=-Dhive.metastore.uris=thrift:// > > >> > > > > > appserver.hadoop.staging.qutics.com:10002 > > >> > > > > > export JAVA_HOME=$(readlink -f /usr/bin/java | sed > > >> "s:bin/java::") > > >> > > > > > exec bin/pig -Dpig.additional.jars=$PIG_CLASSPATH "$@" > > >> > > > > > > > >> > > > > > Any clues? > > >> > > > > > Thank you! > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
