Hi
I already install the lzo, and from the log it sould like successful
mapred.compress.map.output true
&&
syslog logs
2012-06-11 21:50:39,749 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to
load native-hadoop library for your platform... using builtin-java classes
where applicable
2012-06-11 21:50:40,247 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=MAP, sessionId=
2012-06-11 21:50:40,352 WARN org.apache.hadoop.conf.Configuration:
/hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a
attempt to override final parameter: dfs.data.dir; Ignoring.
2012-06-11 21:50:40,361 WARN org.apache.hadoop.conf.Configuration:
/hadoop/tmp/mapred/local/taskTracker/root/jobcache/job_201206111226_0680/job.xml:a
attempt to override final parameter: dfs.name.dir; Ignoring.
2012-06-11 21:50:41,118 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 150
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: data buffer =
119537664/149422080
2012-06-11 21:50:41,234 INFO org.apache.hadoop.mapred.MapTask: record buffer =
393216/491520
2012-06-11 21:50:41,276 INFO com.hadoop.compression.lzo.GPLNativeCodeLoader:
Loaded native gpl library
2012-06-11 21:50:41,280 INFO com.hadoop.compression.lzo.LzoCodec: Successfully
loaded & initialized native-lzo library [hadoop-lzo rev
6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8]
but the pig will give me an error
2012-06-11 21:50:31,507 [Thread-4] ERROR
com.hadoop.compression.lzo.GPLNativeCodeLoader - Could not load native gpl
libraryjava.lang.UnsatisfiedLinkError: no gplcompression in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1738) at
java.lang.Runtime.loadLibrary0(Runtime.java:823) at
java.lang.System.loadLibrary(System.java:1028) at
com.hadoop.compression.lzo.GPLNativeCodeLoader.<clinit>(GPLNativeCodeLoader.java:32)
at com.hadoop.compression.lzo.LzoCodec.<clinit>(LzoCodec.java:71)
at java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:247) at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at
org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:109)
at
org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:156)
at org.apache.hadoop.mapreduce.lib.input.
TextInputFormat.isSplitable(TextInputFormat.java:51) at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:254)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:268)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944) at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961) at
org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170) at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880) at
org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:396) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapred.JobCl
ient.submitJob(JobClient.java:807) at
org.apache.hadoop.mapred.jobcontrol.Job.submit(Job.java:378) at
org.apache.hadoop.mapred.jobcontrol.JobControl.startReadyJobs(JobControl.java:247)
at
org.apache.hadoop.mapred.jobcontrol.JobControl.run(JobControl.java:279)
at java.lang.Thread.run(Thread.java:662)2012-06-11 21:50:31,508 [Thread-4]
ERROR com.hadoop.compression.lzo.LzoCodec - Cannot load native-lzo without
native-hadoopAnd the job could run successful, I donot know the reason. Thank
you for your help.Best RegardsMalone
2012-06-11
