Still not sure how that's happening, but glad it's working for you now :-).
-D On Wed, Feb 24, 2010 at 6:15 AM, Alex Parvulescu <[email protected]>wrote: > Hello and sorry for the absence > > I went through a hard drive crash, and a 2 day recovery process. > > I have incredible news... Pig is working again!!!!!!!!!! > > it felt good to finally say it out loud > > I've found the problem. Check this out: > > --------- This is my test script ----------- > > -- java -cp $PIGDIR/pig-0.6.1-dev-core.jar:$HADOOPDIR org.apache.pig.Main > test_lite.pig > > A = load 't.csv' using PigStorage('\t') as (id: long); > store A into 'output/pig' using PigStorage('\t'); > > -------------------- > > I just saved the run instruction in this script to I don't forget it. > Notice the '$PIGDIR' and '$HADOOPDIR' > > It apprears this is the source if the evil > ERROR 2999: Unexpected internal error. Undefined parameter : PIGDIR > replacing this with > -- $whatever > will give you: ERROR 2999: Unexpected internal error. Undefined parameter : > whatever > > Now that it's working again, I'll be back with more issues in no time :) > > Thanks a million to everybody, especially Dmitriy! > > alex > > > On Thu, Feb 18, 2010 at 2:50 AM, Ashutosh Chauhan < > [email protected]> wrote: > > > Alex, > > > > I just downloaded the latest Pig 0.6 Release Candidate from > > http://people.apache.org/~gates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz> > <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/pig-0.6.0.tar.gz > >and > > followed the instructions in "Trying the Release" section as outlined in > > http://people.apache.org/~gates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt<http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt> > <http://people.apache.org/%7Egates/pig-0.6.0-candidate-1/RELEASE_NOTES.txt > >and > > without taking any extra step anywhere in between I am able to get Pig > > running. You may want to do the same thing. If it works that would be > great > > else we will be sure that its some environment problem in your setup and > we > > can take it from there. > > > > Ashutosh > > > > On Wed, Feb 17, 2010 at 13:51, Dmitriy Ryaboy <[email protected]> > wrote: > > > > > Probably unrelated, but I always run from the root directory, not from > > > bin/, > > > when working with an svn version. Also, never had dev vs core issues. > Can > > > you try cding to pig-branch-0.6-take2 and running ./bin/pig , and put > > > pig.jar (not core, not dev) on $PIG_CLASSPATH? > > > > > > How about avoiding the pig script and just running > > > > > > java -cp $PIGDIR/pig.jar:$HADOOP_CONF_DIR org.apache.pig.Main > > > <<yourscript>> ? > > > > > > I grepped through all the code, including the codegenned stuff, and > none > > of > > > it contains the string PIGDIR (except bin/pig of course). > > > > > > At this point, assuming none of the above helps (which I have little > hope > > > it > > > will), the best I can say is set up your eclipse environment and step > > > through pig's main with the debugger. > > > > > > -D > > > > > > On Wed, Feb 17, 2010 at 2:53 AM, Alex Parvulescu > > > <[email protected]>wrote: > > > > > > > Hello > > > > > > > > This is getting tiresome :) > > > > > > > > Today, I tried with a fresh download of pig 0.5 and I get the exact > > same > > > > error. Something somewhere just broke down, and it took my machine > with > > > it. > > > > At this point I have no way to run any kind of pig version. > > > > Needless to say, this is a big problem for me :) > > > > > > > > Another thing I tried is to do a fresh build of the 0.6 branch. Same > > > error. > > > > I even dropped the sym links. > > > > > > > > Back to our debugging session: > > > > > > > > > svn co > > > > > > > > > > http://svn.apache.org/repos/asf/hadoop/pig/branches/branch-0.6pig-branch-0.6-take2 > > > > A pig-branch-0.6-take2/test/bin/test-patch.sh > > > > Checked out external at revision 910901. > > > > > > > > Checked out revision 910900. > > > > > > > > > cd pig-branch-0.6-take2/ > > > > > ant > > > > BUILD SUCCESSFUL > > > > Total time: 26 seconds > > > > > > > > > cd bin/ > > > > > > > > next is to check if all vars are there > > > > > printenv | grep PIG > > > > PIG_HOME=/home/alex/hadoop/pig-branch-0.6-take2 > > > > PIGDIR=/home/alex/hadoop/pig-branch-0.6-take2 > > > > > > > > > printenv | grep HADOOP > > > > HADOOP_HOME=/home/alex/hadoop/hadoop > > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf > > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf > > > > > > > > > pig > > > > Exception in thread "main" java.lang.NoClassDefFoundError: > > > > jline/ConsoleReaderInputStream > > > > Caused by: java.lang.ClassNotFoundException: > > > jline.ConsoleReaderInputStream > > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > > > > at java.security.AccessController.doPrivileged(Native Method) > > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > > > > at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > > > > Could not find the main class: org.apache.pig.Main. Program will > exit. > > > > > > > > I've sent some email to the list about this problem before. A quick > fix > > > is > > > > to copy and rename the build/pig-0.6.1-dev.jar. Here goes: > > > > > cp ../build/pig-0.6.1-dev.jar ../pig-0.6.1-dev-core.jar > > > > > > > > > pig > > > > 2010-02-17 11:48:52,843 [main] INFO org.apache.pig.Main - Logging > > error > > > > messages to: > > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403732842.log > > > > 2010-02-17 11:48:53,175 [main] INFO > > > > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - > > > Connecting > > > > to hadoop file system at: file:/// > > > > grunt> > > > > > > > > Now at least it will start. > > > > > > > > Getting to the initial issue: > > > > > > > > the '-f' option: > > > > > pig -f /home/alex/hadoop/test/test_lite.pig > > > > 2010-02-17 11:49:28,988 [main] INFO org.apache.pig.Main - Logging > > error > > > > messages to: > > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log > > > > 2010-02-17 11:49:29,012 [main] ERROR org.apache.pig.Main - ERROR > 2999: > > > > Unexpected internal error. Undefined parameter : PIGDIR > > > > Details at logfile: > > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/pig_1266403768987.log > > > > > > > > the '- secretDebugCmd' option > > > > > > > > pig -secretDebugCmd -f /home/alex/hadoop/test/test_lite.pig > > > > dry run: > > > > /usr/lib/jvm/java-6-sun/bin/java -Xmx1000m > > > > -Dpig.log.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/../logs > > > > -Dpig.log.file=pig.log > > > > -Dpig.home.dir=/home/alex/hadoop/pig-branch-0.6-take2/bin/.. > > > > -Dpig.root.logger=INFO,console,DRFA -classpath > > > > > > > > > > > > > > /home/alex/hadoop/pig-branch-0.6-take2/bin/../conf:/usr/lib/jvm/java-6-sun/lib/tools.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/test/classes:/home/alex/hadoop/pig-branch-0.6-take2/bin/../pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../build/pig-0.6.1-dev-core.jar:/home/alex/hadoop/pig-branch-0.6-take2/bin/../lib/hadoop20.jar > > > > org.apache.pig.Main -f /home/alex/hadoop/test/test_lite.pig > > > > > > > > I did not change the scripts, and I get the same error. > > > > > > > > This is really killing me :) > > > > > > > > thanks, > > > > alex > > > > > > > > > > > > > > > > On Tue, Feb 16, 2010 at 6:08 PM, Dmitriy Ryaboy <[email protected]> > > > > wrote: > > > > > > > > > Hm, nothing jumps out. Definitely the error you are getting > indicates > > > > that > > > > > somehow the preprocessor is trying to substitute a variable called > > > PIGDIR > > > > > in > > > > > your script. Which is odd. Does the same thing happen if you try > > > bin/pig > > > > -f > > > > > test_lite.pig? If yes, try running with -secretDebugCmd (shh, it's > > > > secret), > > > > > and sending along the output. Why are you using pig-core instead of > > > > pig.jar > > > > > that ant should be generating for you when you build? Where did you > > get > > > > it, > > > > > how did you build it, and what's the output of cksum on it? > > > > > The fact that now your 0.5 is broken too makes me think that maybe > > your > > > > > symlinks are messed up. Surely a change to one jar shouldn't be > > > affecting > > > > a > > > > > totally unrelated jar in a directory the first jar doesn't know > > about. > > > > > > > > > > Sorry about the barrage of questions, I am just a bit dumbfounded > > about > > > > how > > > > > this could even begin to start happening. Any preprocessor experts > > > > around? > > > > > > > > > > -D > > > > > > > > > > > > > > > On Tue, Feb 16, 2010 at 3:37 AM, Alex Parvulescu > > > > > <[email protected]>wrote: > > > > > > > > > > > Hello > > > > > > > > > > > > And thanks again for all your help > > > > > > > > > > > > I have a symbolic link /home/alex/hadoop/pig which points to > > > > > > /home/alex/hadoop/pig-branch-0.6-ro - this is a checkout and > build > > of > > > > the > > > > > > 0.6 branch. > > > > > > I'm running the script from /home/alex/hadoop/test. > > > > > > > > > > > > I didn't touch pig.properties, also I'm running with the default > > pig > > > > > > script. > > > > > > > > > > > > The only thing I did is I copied the pig/build/pig-0.6.0-dev.jar > > > > > > to pig/pig-0.6.0-dev-core.jar. Because otherwise it would not > > > > > > work(Exception > > > > > > in thread "main" java.lang.NoClassDefFoundError: > > > org/apache/pig/Main). > > > > > > > > > > > > I'm running with a fresh 0.6 build and still no luck. > > > > > > > > > > > > The info you reqested: > > > > > > > > > > > > a...@alex-desktop:~/hadoop/test$ pwd > > > > > > /home/alex/hadoop/test > > > > > > a...@alex-desktop:~/hadoop/test$ printenv | grep JAVA > > > > > > JAVA_HOME=/usr/lib/jvm/java-6-sun > > > > > > a...@alex-desktop:~/hadoop/test$ printenv | grep PIG > > > > > > PIGDIR=/home/alex/hadoop/pig > > > > > > a...@alex-desktop:~/hadoop/test$ printenv | grep HADOOP > > > > > > HADOOP_HOME=/home/alex/hadoop/hadoop > > > > > > HADOOP_CONF_DIR=/home/alex/hadoop/hadoop/conf > > > > > > HADOOPDIR=/home/alex/hadoop/hadoop/conf > > > > > > > > > > > > a...@alex-desktop:~/hadoop/test$ cat > > > > > > /home/alex/hadoop/pig/conf/pig.properties > > > > > > # Pig configuration file. All values can be overwritten by > command > > > line > > > > > > arguments. > > > > > > # see bin/pig -help > > > > > > > > > > > > # log4jconf log4j configuration file > > > > > > # log4jconf=./conf/log4j.properties > > > > > > > > > > > > # brief logging (no timestamps) > > > > > > brief=false > > > > > > > > > > > > # clustername, name of the hadoop jobtracker. If no port is > defined > > > > port > > > > > > 50020 will be used. > > > > > > #cluster > > > > > > > > > > > > #debug level, INFO is default > > > > > > debug=INFO > > > > > > > > > > > > # a file that contains pig script > > > > > > #file= > > > > > > > > > > > > # load jarfile, colon separated > > > > > > #jar= > > > > > > > > > > > > #verbose print all log messages to screen (default to print only > > INFO > > > > and > > > > > > above to screen) > > > > > > verbose=false > > > > > > > > > > > > #exectype local|mapreduce, mapreduce is default > > > > > > #exectype=mapreduce > > > > > > # hod realted properties > > > > > > #ssh.gateway > > > > > > #hod.expect.root > > > > > > #hod.expect.uselatest > > > > > > #hod.command > > > > > > #hod.config.dir > > > > > > #hod.param > > > > > > > > > > > > > > > > > > #Do not spill temp files smaller than this size (bytes) > > > > > > pig.spill.size.threshold=5000000 > > > > > > #EXPERIMENT: Activate garbage collection when spilling a file > > bigger > > > > than > > > > > > this size (bytes) > > > > > > #This should help reduce the number of files being spilled. > > > > > > pig.spill.gc.activation.size=40000000 > > > > > > > > > > > > > > > > > > ###################### > > > > > > # Everything below this line is Yahoo specific. Note that I've > > made > > > > > > # (almost) no changes to the lines above to make merging in from > > > Apache > > > > > > # easier. Any values I don't want from above I override below. > > > > > > # > > > > > > # This file is configured for use with HOD on the production > > > clusters. > > > > > If > > > > > > you > > > > > > # want to run pig with a static cluster you will need to remove > > > > > everything > > > > > > # below this line and set the cluster value (above) to the > > > > > > # hostname and port of your job tracker. > > > > > > > > > > > > exectype=mapreduce > > > > > > > > > > > > hod.config.dir=/export/crawlspace/kryptonite/hod/current/conf > > > > > > hod.server=local > > > > > > > > > > > > cluster.domain=inktomisearch.com > > > > > > > > > > > > log.file= > > > > > > > > > > > > yinst.cluster=kryptonite > > > > > > > > > > > > > > > > > > > > > > > > And now boom! > > > > > > > > > > > > java -cp $PIGDIR/pig-0.6.0-dev-core.jar:$HADOOPDIR > > > org.apache.pig.Main > > > > > > test_lite.pig > > > > > > > > > > > > > > > > > > 2010-02-16 12:27:17,283 [main] INFO org.apache.pig.Main - > Logging > > > > error > > > > > > messages to: /home/alex/hadoop/test/pig_1266319637282.log > > > > > > 2010-02-16 12:27:17,303 [main] ERROR org.apache.pig.Main - ERROR > > > 2999: > > > > > > Unexpected internal error. Undefined parameter : PIGDIR > > > > > > Details at logfile: /home/alex/hadoop/test/pig_1266319637282.log > > > > > > > > > > > > The log file: > > > > > > a...@alex-desktop:~/hadoop/test$ cat pig_1266319637282.log > > > > > > Error before Pig is launched > > > > > > ---------------------------- > > > > > > ERROR 2999: Unexpected internal error. Undefined parameter : > PIGDIR > > > > > > > > > > > > java.lang.RuntimeException: Undefined parameter : PIGDIR > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:232) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106) > > > > > > at > > > > > > > > > > > > > > > > > > > > > > > > > > > org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86) > > > > > > at org.apache.pig.Main.runParamPreprocessor(Main.java:515) > > > > > > at org.apache.pig.Main.main(Main.java:366) > > > > > > > > > > > > > > > > > > > > > > > > > > > ================================================================================ > > > > > > > > > > > > > > > > > > The script: 'A = load '/home/alex/hadoop/test/t.csv' using > > > > > PigStorage('\t') > > > > > > as (id: long); ' > > > > > > The file: '1' . > > > > > > Simple enough :) > > > > > > > > > > > > I'm all out of ideas. It seems that even 0.5 is broken now. I > can't > > > > start > > > > > > anything as scripts. If I go with manual processing in grunt > (line > > by > > > > > line) > > > > > > it's all good on any version. > > > > > > > > > > > > thanks, > > > > > > alex > > > > > > > > > > > > > > > > > > On Tue, Feb 16, 2010 at 9:58 AM, Dmitriy Ryaboy < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Pig starts, the error you are getting is from inside pig.Main . > > > > > > > I think something is getting messed up in your environment > > because > > > > you > > > > > > > are juggling too many different versions of pig (granted, I > have > > 3 > > > or > > > > > > > 4 in various stages of development on my laptop most of the > time, > > > and > > > > > > > haven't had your problems. But then neither of them has a > hacked > > > > > > > bin/pig , with the exception of the cloudera one...). I've > never > > > > tried > > > > > > > running multiple ant tests, either. There's a short "sanity" > > > version > > > > > > > of tests, ant test-commit, that runs in under 10 minutes. You > > might > > > > > > > want to try that if you are not doing things like changing join > > > > > > > implementations or messing with the optimizer. > > > > > > > > > > > > > > Let's do this: send a full trace of what you are doing and what > > > your > > > > > > > environment looks like. Something like > > > > > > > > > > > > > > pwd > > > > > > > printenv | grep JAVA > > > > > > > printenv | grep PIG > > > > > > > printenv | grep HADOOP > > > > > > > cat conf/pig.properties > > > > > > > java -cp ........ > > > > > > > <boom!> > > > > > > > > > > > > > > -D > > > > > > > > > > > > > > > > > > > > > On Tue, Feb 16, 2010 at 12:39 AM, Alex Parvulescu > > > > > > > <[email protected]> wrote: > > > > > > > > Hello, > > > > > > > > > > > > > > > > sorry for the delay, but I wanted to build from the source > > again > > > > just > > > > > > to > > > > > > > > make sure. > > > > > > > > > > > > > > > > The script is like this: > > > > > > > > > > > > > > > > 'A = load '/home/alex/hadoop/reviews/r.csv' using > > > PigStorage('\t') > > > > as > > > > > > > (id: > > > > > > > > long, hid: long, locale: chararray, r1: int, r2: int, r3: > int, > > > r4: > > > > > > int); > > > > > > > ' > > > > > > > > > > > > > > > > That's it. My guess is that Pig doesn't even start. Do you > > think > > > I > > > > > need > > > > > > > to > > > > > > > > change something in the properties file? I'm not sure anymore > > and > > > I > > > > > > don't > > > > > > > > have a lot of luck going through the sources. > > > > > > > > > > > > > > > > And another thing, I tried running 'ant test' for both > > 0.6-branch > > > > and > > > > > > > trunk > > > > > > > > at the same time (because they take a very long time) and > both > > > test > > > > > > > scripts > > > > > > > > failed. I've switched to running them one after the other and > > > they > > > > > are > > > > > > > fine. > > > > > > > > Do you think that is ok? > > > > > > > > > > > > > > > > thanks for your time, > > > > > > > > alex > > > > > > > > > > > > > > > > On Fri, Feb 12, 2010 at 6:37 PM, Dmitriy Ryaboy < > > > > [email protected]> > > > > > > > wrote: > > > > > > > > > > > > > > > >> what does your script1-hadoop.pig look like? > > > > > > > >> > > > > > > > >> The error you are getting happens when the pig preprocessor > > > can't > > > > > > > >> substitute Pig variables (the stuff you specify with -param > > and > > > > > > > >> %default, etc). Do you have $PIGDIR in your script > somewhere? > > > > > > > >> > > > > > > > >> -D > > > > > > > >> > > > > > > > >> On Fri, Feb 12, 2010 at 6:51 AM, Alex Parvulescu > > > > > > > >> <[email protected]> wrote: > > > > > > > >> > Hello, > > > > > > > >> > > > > > > > > >> > I seem to have broken my Pig install, and I don't know > where > > > to > > > > > > look. > > > > > > > >> > > > > > > > > >> > If I use directly the script (grunt) everything works ok, > > but > > > > > every > > > > > > > time > > > > > > > >> I > > > > > > > >> > try to run a pig script: 'java -cp > > > > > $PIGDIR/pig.jar:$HADOOPSITEPATH > > > > > > > >> > org.apache.pig.Main script1-hadoop.pig' > > > > > > > >> > > > > > > > > >> > I get this nice error: [main] ERROR org.apache.pig.Main - > > > ERROR > > > > > > 2999: > > > > > > > >> > Unexpected internal error. Undefined parameter : PIGDIR > > > > > > > >> > > > > > > > > >> > Obviously I have the PIGDIR var set: > > > > > > > >> >> echo $PIGDIR > > > > > > > >> >> /home/alex/hadoop/pig > > > > > > > >> > > > > > > > > >> > This is something that I did, as I have used 0.5 and 0.6 > and > > a > > > > > > patched > > > > > > > >> > version of 0.6 in parallel, but I can't figure out where > to > > > > look. > > > > > > Any > > > > > > > >> > version I try to start now, gives the same error. > > > > > > > >> > > > > > > > > >> > any help would be greatly appreciated! > > > > > > > >> > > > > > > > > >> > alex > > > > > > > >> > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
