You will need either: set HADOOP_HOME, or use -x local
On Tue, Apr 24, 2012 at 12:52 AM, Prashant Kommireddi <[email protected]> wrote: > Aaah I think that clue helped. This was my setup: > > HADOOP_HOME was set in .bashrc > > In my wrapper script I unset HADOOP_HOME for pig to pick up the bundled > version 0.20.2. This works for 0.9.1. > > 0.10.0 however picks up HADOOP_HOME though I unset it in my wrapper script. > Ofcourse, it worked after removing HADOOP_HOME from .bashrc altogether. > Though now I see an error: ERROR 2999: Unexpected internal error. Failed to > create DataStorage > > This is the output of secretDebugCmd > > Cannot find local hadoop installation, using bundled hadoop 20.2 > dry run: > /home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/bin/java -Xmx1000m > -Djava.library.path=/home/pkommireddi/dev/tools/Linux/hadoop/hadoop-0.20.2/lib/native/Linux-amd64-64 > -Dpig.log.dir=/home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/../logs > -Dpig.log.file=pig.log > -Dpig.home.dir=/home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/.. > -classpath > /home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/../conf:/home/pkommireddi/dev/tools/Linux/jdk/jdk1.6.0_21_x64/lib/tools.jar:/home/pkommireddi/dev/apps/gridforce/main/hadoop/conf/dev:/home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/../lib/automaton.jar:/home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/../lib/jython-2.5.0.jar:/home/pkommireddi/dev/tools/Linux/hadoop/pig-0.10.0/bin/../pig-0.10.0.jar > org.apache.pig.Main > > > > > > > On Tue, Apr 24, 2012 at 12:09 AM, Daniel Dai <[email protected]> wrote: > >> Do you have HADOOP_HOME? Both HADOOP_CONF_DIR/PIG_CLASSPATH should >> work, can you use bin/pig -secretDebugCmd to check hadoop command >> line? >> >> On Mon, Apr 23, 2012 at 11:32 PM, Prashant Kommireddi >> <[email protected]> wrote: >> > Thanks Dmitriy, that works. But I am wondering why the behavior is >> > different from the previous versions. >> > >> > Difference I see in bin/pig is (0.10.0 vs 0.9.1) >> > >> >> # add HADOOP_CONF_DIR >> >> if [ "$HADOOP_CONF_DIR" != "" ]; then >> >> CLASSPATH=${CLASSPATH}:${HADOOP_CONF_DIR} >> >> fi >> > >> > AFAIK, this should not affect it - all it's doing is adding the conf dir >> to >> > the classpath which I was doing earlier through PIG_CLASSPATH in my >> wrapper >> > script. >> > >> > The issue here is that certain properties are not same between client >> > machine and remote cluster, for eg JAVA_HOME. Since pig is client side it >> > made sense for Pig to not pick up any cluster properties from >> > "hadoop-env.sh". I am not sure what the change here is that's now causing >> > it to be picked up. >> > >> > >> > >> > On Mon, Apr 23, 2012 at 9:14 PM, Dmitriy Ryaboy <[email protected]> >> wrote: >> > >> >> pig.sh understands the arguments below -- try setting HADOOP_CONF_DIR? >> >> >> >> # Environment Variables >> >> # >> >> # JAVA_HOME The java implementation to use. >> >> Overrides JAVA_HOME. >> >> # >> >> # PIG_CLASSPATH Extra Java CLASSPATH entries. >> >> # >> >> # HADOOP_HOME/HADOOP_PREFIX Environment >> >> HADOOP_HOME/HADOOP_PREFIX(0.20.205) >> >> # >> >> # HADOOP_CONF_DIR Hadoop conf dir >> >> # >> >> # PIG_HEAPSIZE The maximum amount of heap to use, in MB. >> >> # Default is 1000. >> >> # >> >> # PIG_OPTS Extra Java runtime options. >> >> # >> >> # PIG_CONF_DIR Alternate conf dir. Default is ${PIG_HOME}/conf. >> >> # >> >> # HBASE_CONF_DIR - Optionally, the HBase configuration to run >> against >> >> # when using HBaseStorage >> >> >> >> >> >> >> >> On Mon, Apr 23, 2012 at 8:45 PM, Prashant Kommireddi >> >> <[email protected]> wrote: >> >> > I have a wrapper script to switch between Pig versions and clusters. >> >> > >> >> > export PIG_HOME=$HOME/tools/Linux/hadoop/pig-$PIG_VERSION >> >> > export JAVA_HOME=$HOME/tools/Linux/jdk/jdk$JAVA_VERSION/ >> >> > export >> >> PIG_CLASSPATH=$HOME/apps/gridforce/main/hadoop/conf/$HADOOP_CLUSTER >> >> > >> >> > HADOOP_CLUSTER contains the hadoop configs (endpoints) for the >> cluster I >> >> > want to point to. >> >> > >> >> > And then I do this to start pig. >> >> > >> >> > $PIG_HOME/bin/pig >> >> > >> >> > This works with previous versions pig-0.8.0 and pig-0.9.1. However, >> >> > pig-0.10.0 fails to pick up the right classpath. >> >> > >> >> > $ ./pig.sh >> >> > .. >> >> > ..... >> >> > ...... >> >> > 2012-04-23 20:42:35,340 [main] INFO >> >> > org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - >> >> Connecting >> >> > to hadoop file system at: file:/// >> >> > >> >> > Is this something with my script, or may be the new version (0.10.0)? >> >> > >> >> > -Prashant >> >> > >> >> > >> >> > On Mon, Apr 23, 2012 at 8:30 PM, Dmitriy Ryaboy <[email protected]> >> >> wrote: >> >> > >> >> >> Uh, actually, one of the test-commit tests failed in my environment. >> >> >> >> >> >> In TestPigServer: >> >> >> >> >> >> Testcase: testDefaultPigProperties took 0.033 sec >> >> >> Caused an ERROR >> >> >> null >> >> >> java.lang.NullPointerException >> >> >> at >> >> >> >> >> >> org.apache.pig.test.TestPigServer.testDefaultPigProperties(TestPigServer.java:895) >> >> >> >> >> >> Something about my environment? >> >> >> >> >> >> D >> >> >> >> >> >> >> >> >> On Mon, Apr 23, 2012 at 6:36 PM, Dmitriy Ryaboy <[email protected]> >> >> >> wrote: >> >> >> > +1 >> >> >> > >> >> >> > >> >> >> > Verified several jobs using Elephant-Bird loaders. >> >> >> > Tested correctness with pig.exec.mapPartAgg both true and false. >> >> >> > Verified license. >> >> >> > Verified release notes. >> >> >> > Ran test-commit >> >> >> > >> >> >> > D >> >> >> > >> >> >> > On Sat, Apr 21, 2012 at 12:27 PM, Daniel Dai < >> [email protected]> >> >> >> wrote: >> >> >> >> We should do sanity check of the package, such as unit tests, e2e >> >> >> >> tests, piggybank tests, package integrity, package signature, >> >> license, >> >> >> >> etc. However, if we find a new bug, usually we will push it to the >> >> >> >> next release at this stage unless it is a critical one. >> >> >> >> >> >> >> >> Thanks, >> >> >> >> Daniel >> >> >> >> >> >> >> >> On Sat, Apr 21, 2012 at 12:48 AM, Prashant Kommireddi >> >> >> >> <[email protected]> wrote: >> >> >> >>> Hi Daniel, >> >> >> >>> >> >> >> >>> What is required other than running the regular tests for testing >> >> >> release >> >> >> >>> candidate? I can think of running a few existing scripts against >> >> >> candidate >> >> >> >>> build and making sure outputs look fine. >> >> >> >>> >> >> >> >>> Thanks, >> >> >> >>> Prashant >> >> >> >>> >> >> >> >>> On Fri, Apr 20, 2012 at 12:39 AM, Daniel Dai < >> [email protected] >> >> > >> >> >> wrote: >> >> >> >>> >> >> >> >>>> Hi, >> >> >> >>>> >> >> >> >>>> I have created a candidate build for Pig 0.10.0. >> >> >> >>>> >> >> >> >>>> Keys used to sign the release are available at >> >> >> >>>> http://svn.apache.org/viewvc/pig/trunk/KEYS?view=markup. >> >> >> >>>> >> >> >> >>>> Please download, test, and try it out: >> >> >> >>>> >> >> >> >>>> http://people.apache.org/~daijy/pig-0.10.0-candidate-0/ >> >> >> >>>> >> >> >> >>>> Should we release this? Vote closes on next Tuesday, Apr 24th. >> >> >> >>>> >> >> >> >>>> Daniel >> >> >> >>>> >> >> >> >> >> >>
