Reply inline! Regards, Dhaval
----- Original Message ----- From: Cheolsoo Park <[email protected]> To: [email protected]; Dhaval Shah <[email protected]> Cc: Sent: Wednesday, 10 October 2012 4:55 PM Subject: Re: Error with Pig (CDH4.0.0) Hi Dhaval, Thank you very much for sharing your analysis! Your explanation definitely provides more insights. :-) If you don't mind, I'd like to clarify a couple of things. I am just trying to see if there's something to be fixed in terms of Pig packaging: >> the root issue was that the pig libraries needed to be present in the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh.. I am not sure if missing dependency libraries is the root cause. Given the following error in your call stack, I believe that you have a different version of antlr in classpath. ---Dhaval -> Yes I did have Antlr 2.7.7 and 3.0.1 in the classpath from Mahout 0.6.. However, I still do have it in my classpath (and it appears earlier in the classpath than Pig) and it still works for me.. Not sure what the actual issue is but something to think about.. Also, I saw that pig jars have antlr bundled.. So technically those should take precedence anyways unless its being called in a different way.. java.lang.NoSuchFieldError: name > at org.apache.pig.parser.QueryParserStringStream.<init> > (QueryParserStringStream.java:32) If you look at QueryParserStringStream.java, it extends ANTLRStringStream, and Pig 0.9 is compiled against antlr 3.4. Now if antlr is missing, you should get ClassNotFoundError not NoSuchFiledError. Only possible reason that I can think of is that there is a different version of antlr in classpath at run-time. In fact, there was a similar discussion on this mailing list a while ago (Please note that he is reporting the same call stack as yours): http://mail-archives.apache.org/mod_mbox/pig-user/201206.mbox/%3CCAOkn+GjPCgY1i+-zfNWF9cKz+m8feRj=6rk1shss293oqec...@mail.gmail.com%3E Antlr is a common tool, so it's possible that it's installed by other tools such as Hive. >> I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries in the shell and then start pig but that did not help either.. Assuming that a wrong version of antlr is present in classpath, this makes perfect sense because PIG_CLASSPATH adds libraries to the end of CLASPATH: ---Dhaval --> I think PIG_CLASSPATH libraries should be added to the front to take precedence since we are calling pig and expressing that we want to use pig and not anything else # add user-specified CLASSPATH if [ "$PIG_CLASSPATH" != "" ]; then CLASSPATH=${CLASSPATH}:${PIG_CLASSPATH} fi That is, as long as the wrong antlr is present before the correct one, the wrong one be picked up at run-time. >> 2. The error message is not super helpful.. If libraries are missing, the pig shell/grunt should not open at all.. However, in my case, it did start up and then the error message was in no ways intuitive or pointing to the root issue.. I agree with you that the error message is not very intuitive here. But errors caused by dependency libraries can be only caught when Pig makes calls to methods of those libraries at run-time. Furthermore, if there are two different versions of the same library in classpath, the root cause can be more subtle. The challenge is that a Hadoop distribution such as CDH bundles many sub-projects, and they often depend on different versions of the same libraries. We do our best to harmonize all the versions of dependency libraries across the platform, but I admit that it is not always perfect. ---Dhaval --> True I understand that issue and I guess the only thing we could do is update the pig executable here to exclude the known conflicts which in my opinion is a reasonable thing to do Thanks, Cheolsoo On Wed, Oct 10, 2012 at 7:57 AM, Dhaval Shah <[email protected]>wrote: > Alright I was eventually able to get the issue resolved.. For everyone's > benefit, the root issue was that the pig libraries needed to be present in > the HADOOP_CLASSPATH and it needs to be specifically set in hadoop_env.sh.. > I tried setting PIG_CLASSPATH and HADOOP_CLASSPATH to have those libraries > in the shell and then start pig but that did not help either.. Setting that > explicitly in hadoop_env.sh did the trick.. > > As I see there are 2 issues here: > 1. Somehow the pig executable seems broken.. As in setting > HADOOP_CLASSPATH on the shell and then running pig should have the same > effect as setting it in hadoop_env.sh.. Also having a * in the > PIG_CLASSPATH/HADOOP_CLASSPATH (when set in the shell) makes the pig > executable set HADOOP_CLASSPATH to empty before it actually runs the hadoop > jar command inside the pig script.. In CDH3, I never needed to set the > classpath to include pig libraries.. The pig executable did that for > me..(and I am never overriding HADOOP_CLASSPATH.. I only append to it) > 2. The error message is not super helpful.. If libraries are missing, the > pig shell/grunt should not open at all.. However, in my case, it did start > up and then the error message was in no ways intuitive or pointing to the > root issue.. > > Also, I don't think this is a legacy pig issue as we tried the newer > version 0.10 and it came up with the same issue.. A second issue that did > pop up after putting the pig libraries in the classpath was with mapreduce > framework being selected as classic and it turned out that the MRv1 > libraries needed to be explicitly specified in the HADOOP_CLASSPATH > (without that entry Sqoop, Java MR jobs work fine but Pig did not for > whatever reason) > > Anyways thanks everyone for the help > > Regards, > Dhaval > > > ----- Original Message ----- > From: Russell Jurney <[email protected]> > To: "[email protected]" <[email protected]> > Cc: > Sent: Tuesday, 9 October 2012 7:34 PM > Subject: Re: Error with Pig (CDH4.0.0) > > I've often wondered - as we see a lot of these legacy pig issues in > CDH on this list - is it hard to upgrade Pig on CDH by downloading the > latest stable release of pig, unpacking the .tgz and running it? Is > upgrading Pig on CDH as simple as wget, like on Apache Hadoop and > others, or is it somehow more complex? Pig is after all a client-side > tool. > > Russell Jurney http://datasyndrome.com > > On Oct 9, 2012, at 4:15 PM, Cheolsoo Park <[email protected]> wrote: > > > Hi Dhaval, > > > > That certainly works for me. > > > > How did you upgrade CDH? Did you install it via RPMs? Did you completely > > uninstall CDH3u3 before installing CDH4? > > > > It sounds to me like a CDH-upgrade issue rather than Pig issue. Can you > > please provide steps that you took to upgrade CDH? > > > > Thanks, > > Cheolsoo > > > > On Tue, Oct 9, 2012 at 4:02 PM, Dhaval Shah <[email protected] > >wrote: > > > >> Thanks for getting back Cheolsoo.. > >> > >> All I am trying to do is run the pig shell/grunt and do this: > >> p = LOAD 'file_name'; > >> > >> It comes back with the exception mentioned below.. > >> > >> Regards, > >> Dhaval > >> > >> > >> ----- Original Message ----- > >> From: Cheolsoo Park <[email protected]> > >> To: [email protected]; Dhaval Shah <[email protected]> > >> Cc: > >> Sent: Tuesday, 9 October 2012 6:57 PM > >> Subject: Re: Error with Pig (CDH4.0.0) > >> > >> Hi Dhaval, > >> > >> CDH3u3 includes Pig 0.8, and CDH4.0.0 includes Pig 0.9. There were > >> some incompatibilities introduced between two version. > >> > >> > >> > https://cwiki.apache.org/confluence/display/PIG/Pig+0.9+Backward+Compatibility > >> > >> To pin down the exact cause, I'd like to reproduce your error. Would > >> mind providing an example script that generates the exception? > >> > >> Thanks, > >> Cheolsoo > >> > >> On Tue, Oct 9, 2012 at 3:29 PM, Dhaval Shah < > [email protected] > >>> wrote: > >> > >>> Hi everyone.. We just upgraded to CDH4.0.0 and are seeing a very weird > >>> issue with Pig.. Everytime I try to run a LOAD command, it dies with > the > >>> following exception: > >>> > >>> ERROR 2998: Unhandled internal error. name > >>> > >>> java.lang.NoSuchFieldError: name > >>> at > >>> > >> > org.apache.pig.parser.QueryParserStringStream.<init>(QueryParserStringStream.java:32) > >>> at > >>> > >> > org.apache.pig.parser.QueryParserDriver.tokenize(QueryParserDriver.java:194) > >>> at > >>> > org.apache.pig.parser.QueryParserDriver.parse(QueryParserDriver.java:162) > >>> at > >>> org.apache.pig.PigServer$Graph.validateQuery(PigServer.java:1609) > >>> at > >>> org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1582) > >>> at org.apache.pig.PigServer.registerQuery(PigServer.java:584) > >>> at > >>> org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:967) > >>> at > >>> > >> > org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:386) > >>> at > >>> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189) > >>> at > >>> > >> > org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165) > >>> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69) > >>> at org.apache.pig.Main.run(Main.java:495) > >>> at org.apache.pig.Main.main(Main.java:111) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at > >>> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > >>> at > >>> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >>> at java.lang.reflect.Method.invoke(Method.java:597) > >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:208) > >>> > >>> The same script/command worked fine with CDH3U3.. Did any APIs change? > Or > >>> is this a bug? > >>> Regards, > >>> Dhaval > >>> > >> > >> > >
