Look at http://incubator.apache.org/hcatalog/docs/r0.4.0/loadstore.htmlread section "Running Pig with HCatalog"
export HADOOP_HOME=<path_to_hadoop_install> export HCAT_HOME=<path_to_hcat_install> export PIG_CLASSPATH=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar:$HIVE_HOME/lib/hive-metastore-0.9.0.jar: $HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar:$HIVE_HOME/lib/libfb303-0.7.0.jar: $HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/conf:$HADOOP_HOME/conf:$HIVE_HOME/lib/slf4j-api-1.6.1.jar export PIG_OPTS=-Dhive.metastore.uris=thrift://<hostname>:<port> /* invoke pig */ <path_to_pig_install>/bin/pig -Dpig.additional.jars=$HCAT_HOME/share/hcatalog/hcatalog-0.4.0.jar: $HIVE_HOME/lib/hive-metastore-0.9.0.jar:$HIVE_HOME/lib/libthrift-0.7.0.jar:$HIVE_HOME/lib/hive-exec-0.9.0.jar: $HIVE_HOME/lib/libfb303-0.7.0.jar:$HIVE_HOME/lib/jdo2-api-2.3-ec.jar:$HIVE_HOME/lib/slf4j-api-1.6.1.jar <script.pig> Essentially you need to set PIG_CLASSPATH env variable as well as set -Dpig.additional.jars pointing to same jar files -Dpig,additional.jars=$PIG_CLASSPATH Here is the relevant (and fairly long) threadin the mailing list which will give you an idea on why you need both PIG_CLASSPATH as well as set 'pig,additional.jars' http://mail-archives.apache.org/mod_mbox/incubator-hcatalog-user/201206.mbox/%3CCANSvDjp1m=nuzc=T+Oi1OJmnjkc_SsycTAvn0cJJ16s=brd...@mail.gmail.com%3E Note: I am not using HDP but hopefully above would shed some light on your problem -Agatea On Fri, Aug 24, 2012 at 9:58 AM, Petr Babkin <pbab...@revelytix.com> wrote: > Thank you, Mr. Gates! > > Now pig doesn't complain about hive-site.xml anymore, however it gets me > the same error: > *ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal > error. com/facebook/fb303/FacebookService$Iface* > The log file says: > java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) > at java.lang.ClassLoader.defineClass(ClassLoader.java:615) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) > at java.net.URLClassLoader.access$000(URLClassLoader.java:58) > at java.net.URLClassLoader$1.run(URLClassLoader.java:197) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > at > org.apache.hcatalog.pig.PigHCatUtil.createHiveMetaClient(PigHCatUtil.java:114) > at > org.apache.hcatalog.pig.PigHCatUtil.getTable(PigHCatUtil.java:150) > at > org.apache.hcatalog.pig.HCatLoader.getSchema(HCatLoader.java:151) > at > org.apache.pig.newplan.logical.relational.LOLoad.getSchemaFromMetaData(LOLoad.java:150) > at > org.apache.pig.newplan.logical.relational.LOLoad.getSchema(LOLoad.java:109) > at > org.apache.pig.newplan.logical.visitor.LineageFindRelVisitor.visit(LineageFindRelVisitor.java:100) > at > org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:218) > at > org.apache.pig.newplan.DependencyOrderWalker.walk(DependencyOrderWalker.java:75) > at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50) > at > org.apache.pig.newplan.logical.visitor.CastLineageSetter.<init>(CastLineageSetter.java:57) > at org.apache.pig.PigServer$Graph.compile(PigServer.java:1679) > at > org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1610) > at > org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582) > at org.apache.pig.PigServer.registerQuery(PigServer.java:584) > at > org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:942) > at > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:188) > at > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:164) > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > at org.apache.pig.Main.run(Main.java:495) > at org.apache.pig.Main.main(Main.java:111) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > Caused by: java.lang.ClassNotFoundException: > com.facebook.fb303.FacebookService$Iface > at java.net.URLClassLoader$1.run(URLClassLoader.java:202) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:190) > at java.lang.ClassLoader.loadClass(ClassLoader.java:306) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > at java.lang.ClassLoader.loadClass(ClassLoader.java:247) > ... 38 more > > It's quite surprising how I'm running into these issue having launched HDP > from Hortonworks' pre-configured VM. I just can't imagine a place where I > could have messed up with the configuration. > > Regards, > Petr Babkin. > > On Fri, Aug 24, 2012 at 12:44 PM, Alan Gates <ga...@hortonworks.com>wrote: > >> Also your PIG_CLASSPATH has to point to the directory that hive-site.xml >> is in, not to the hive-site.xml file itself. >> >> Alan. >> >> On Aug 24, 2012, at 9:10 AM, Petr Babkin wrote: >> >> > Thanks, Aniket and Vandana, >> > >> > after adding the jars to PIG_CLASSPATH, I'm getting: >> > >> > WARN org.apache.hadoop.hive.conf.HiveConf - hive-site.xml not found on >> CLASSPATH >> > ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2998: Unhandled internal >> error. com/facebook/fb303/FacebookService$Iface >> > >> > though my PIG_CLASSPATH has /etc/hive/conf.dist/hive-site.xml added. >> > >> > Vandana, could you please be more specific on what HCatalog >> configuration needs to be done? >> > >> > Many thanks! >> > >> > Regards, >> > Petr Babkin >> > >> > On Thu, Aug 23, 2012 at 7:02 PM, Aniket Mokashi <aniket...@gmail.com> >> wrote: >> > 1. Make sure HCatalog jar is in the class path (named hcatalog*.jar) >> > 2. Make sure your pig is running with >> -Dudf.import.list=org.apache.hcatalog.pig. This is how pig finds the >> package prefix for HCatLoader >> > >> > Workaround for 2 is- in your pig script- replace HCatLoader by >> org.apache.hcatalog.pig.HCatLoader. >> > >> > I assume HDP is doing all that already. So, I suspect something else is >> responsible here. >> > >> > Thanks, >> > Aniket >> > >> > On Thu, Aug 23, 2012 at 3:47 PM, Vandana Ayyalasomayajula < >> avand...@yahoo-inc.com> wrote: >> > Hi Petr, >> > >> > I am thinking that this issue is related to class path. You need to >> have the HCatalog, hive jars in your PIG_CLASSPATH env variable. >> > >> > From a previous post in the user list, the following this are required >> in the class path. >> > >> > Minimum set- >> > /share/hcatalog/hcatalog*.jar: >> > /share/hcatalog/lib/hive-metastore*.jar: >> > /share/hcatalog/lib/thrift-fb303-*.jar: >> > /share/hcatalog/lib/libthrift*.jar: >> > /share/hcatalog/lib/hive-exec-*.jar >> > +HCatalog configuration (hive-site et al)… >> > >> > Thanks >> > Vandana >> > >> > >> > On Aug 23, 2012, at 12:49 PM, Petr Babkin wrote: >> > >> >> I am getting the following error when trying to load the table from >> HCatalog into pig: >> >> >> >> ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1070: Could not resolve >> org.apache.hcatalog.pig.HCatLoader using imports: [, >> org.apache.pig.builtin., org.apache.pig.impl.builtin.] >> >> >> >> This is all running on HDP, so I am surprised why the resolution >> fails. Googling around this issue didn't bring any result. >> >> >> >> Thanks. >> >> >> >> -- >> >> Regards, >> >> Petr Babkin >> >> Ontologist, Revelytix Inc. >> >> >> > >> > Vandana Ayyalasomayajula >> > >> > >> > >> > >> > >> > >> > -- >> > "...:::Aniket:::... Quetzalco@tl" >> > >> > >> > >> > -- >> > Regards, >> > Petr Babkin >> > Ontologist, Revelytix Inc. >> > >> >> > > > -- > Regards, > Petr Babkin > Ontologist, Revelytix Inc. > >
