OK, it seems you have an issue with your classpaths. It may be an idea to make a file called "generateData". Then in this, contain the following lines: ------------------------- export pigjar=$PIGHOME/pig-0.4.0-core.jar export zipfjar=[LOCATION_TO]/sdsuLibJKD14.jar export datagenjar=[LOCATION_TO]/test.jar #(that you have created including the patch) export conf_file=$HADOOP_HOME/conf/hadoop-site.xml export HADOOP_CLASSPATH=$pigjar:$zipfjar:$datagenjar $HADOOP_HOME/bin/hadoop jar $datagenjar org.apache.pig.test.utils.datagen.DataGenerator -conf $conf_file -rows 100 -f generated_data.dat s:8:50:z:0 ----------------------
Providing "echo $HADOOP_CLASSPATH" looks OK, there should be a file generated_data.dat in the same directory as the generateData file, once the generateData file has been executed. Let us know how you get on. Rob Stewart 2009/12/11 Iman E <[email protected]> > Thankyou so much, Rob for your reply. > > Moving the -libjars fixed the error, but I still get this error: > Exception in thread "main" java.lang.ClassNotFoundException: > org.apache.pig.test.utils.datagen.DataGenerator > at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > at java.lang.ClassLoader..loadClass(ClassLoader.java:307) > at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > at java.lang..ClassLoader.loadClassInternal(ClassLoader.java:320) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:247) > at org.apache.hadoop.util.RunJar.main(RunJar.java:149) > > The datagenjar (which is the test.jar that I have generated) has the > org.apache.pig.test.utils.datagen..DataGenerator but still it did not fix > the error. > Thanks > Iman > > > > > ________________________________ > From: Rob Stewart <[email protected]> > To: [email protected] > Sent: Fri, December 11, 2009 2:13:17 PM > Subject: Re: DataGenerator Location > > This is a good point.. the DataGenerator on the Wiki page needs updating. > > In Hadoop 0.20 the -libjars comes *after* the jar file. > > It should say: > hadoop jar $datagenjar > org.apache.pig.test.utils.datagen.DataGenerator</pig/DataGenerator> > -libjars > $zipfjar -conf $conf_file [options] colspec... > > > Iman - Could you try doing this and let us know? > > Thanks, > > Rob Stewart > > > 2009/12/11 Iman E <[email protected]> > > > Hi all, > > Thanks for sharing PigMix with us. > > I do have a related question to this thread. The generate_data.sh > > references test.jar in datagenjar=$PIG_HOME/build/test/classes/test.jar. > > This jar is never generated for me. I generated one out of the test class > > files and hope that it was it. > > My other question, I tried to run the script but I keep getting this > error: > > Exception in thread "main" java.io.IOException: Error opening job jar: > > -libjars > > at org.apache.hadoop.util.RunJar.main(RunJar.java:90) > > Caused by: java.util.zip.ZipException: error in opening zip file > > at java.util.zip.ZipFile.open(Native Method) > > at java.util.zip.ZipFile.<init>(ZipFile.java:114) > > at java.util.jar.JarFile.<init>(JarFile.java:133) > > at java.util.jar.JarFile.<init>(JarFile.java:70) > > at org.apache.hadoop.util.RunJar.main(RunJar.java:88) > > > > I made sure that the sdsuLibJKD12.jar is in place and I was able to > compile > > the tests using it, but still the script will not work! Any suggestions. > > Thanks > > > > > > > > > > ________________________________ > > From: Ashutosh Chauhan <[email protected]> > > To: [email protected] > > Sent: Mon, November 2, 2009 12:01:00 PM > > Subject: Re: DataGenerator Location > > > > > > > > > > > I Have searched through the jar's in both the Pig 0.4.0 and 0.5.0 and > > > cannot find any such classpath. Where can I expect to find this > > > DataGenerator class? > > > > > > > As Alan wrote earlier DataGenerator is in patch > > https://issues.apache.org/jira/browse/PIG-200 You need to download the > > patch > > and build it. > > > > > > > > Finally, that Wiki page refers to specifying the path to $zipfjar: > > > sdsuLibJKD12.jar . Where can I find this jar file? > > > > > > > http://www.eli.sdsu.edu/java-SDSU/ > > > > Hope it helps, > > Ashutosh > > > > > > > > > > > > >
