What's currently on top of trunk requires use of hadoop17.jar instead of
hadoop16.jar.
Alan.
Iván de Prado wrote:
I did ant clean, and tried to recompile with hadoop16.jar. But I got
these compilation errors:
[javac] Compiling 241 source files to /opt/hd1/pig-trunk/build/classes
[javac]
/opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:28:
cannot find symbol
[javac] symbol : class FileOutputFormat
[javac] location: package org.apache.hadoop.mapred
[javac] import org.apache.hadoop.mapred.FileOutputFormat;
[javac] ^
[javac]
/opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/SortPartitioner.java:26:
cannot find symbol
[javac] symbol : class RawComparator
[javac] location: package org.apache.hadoop.io
[javac] import org.apache.hadoop.io.RawComparator;
[javac] ^
[javac]
/opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/SortPartitioner.java:37:
cannot find symbol
[javac] symbol : class RawComparator
[javac] location: class
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.SortPartitioner
[javac] RawComparator comparator;
[javac] ^
[javac]
/opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:48:
cannot find symbol
[javac] symbol : variable FileOutputFormat
[javac] location: class
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat
[javac] Path outputDir = FileOutputFormat.getWorkOutputPath(job);
[javac] ^
[javac]
/opt/hd1/pig-trunk/src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigOutputFormat.java:62:
cannot find symbol
[javac] symbol : variable FileOutputFormat
[javac] location: class
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat
[javac] String parentName =
FileOutputFormat.getOutputPath(job).getName();
Is not Pig compatible with Hadoop 0.16 anymore? Did I do something wrong when
compiling?
I'm using the revision 661633
Iván
El vie, 30-05-2008 a las 13:06 +0200, Iván de Prado escribió:
With the latest version I'm getting an error:
java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/FileOutputFormat
at
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigOutputFormat.getRecordWriter(PigOutputFormat.java:48)
at
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.setupMapPipe(PigMapReduce.java:257)
at
org.apache.pig.backend.hadoop.executionengine.mapreduceExec.PigMapReduce.run(PigMapReduce.java:111)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapred.FileOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:276)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 5 more
What did I do wrong? I'm launching pig using bin/pig script. Before the update, it worked.
Iván de Prado
www.ivanprado.es
El vie, 30-05-2008 a las 20:01 +1000, pi song escribió:
We've already fixed the memory issue introduced in Pig-85. Could you please
update to the latest version and try again?
Pi
On Wed, May 28, 2008 at 9:18 AM, pi song <[EMAIL PROTECTED]> wrote:
This might have nothing to do with Hadoop 0.17 but something else that we
fixed right after it. I'm investigating. Sorry for inconvenience.
FYI,
Pi
On 5/28/08, Tanton Gibbs <[EMAIL PROTECTED]> wrote:
I think you need to increase the amount of memory you give to java.
It looks like it is currently set to 256M. I upped mine to 2G. Of
course it depends on how much ram you have available.
mapred.child.java.opts is the parameter
mine is currently set to 2048M in my hadoop-site.xml file.
For performance reasons, I upped the io.sort.mb parameter. However,
if this is too close to 50% of the total memory, you will get the
Spillable messages.
HTH,
Tanton
