Hi Ted, Please make sure you have version 2.02 of liblzo installed. There's an earlier version in RHEL that doesn't fail properly at the configure stage and I've seen it produce the error you pasted below.
-Todd On Tue, Jan 12, 2010 at 10:15 AM, Ted Yu <yuzhih...@gmail.com> wrote: > I followed http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ > > Package gcc-c++-4.1.2-46.el5_4.1.x86_64 already installed and latest > version > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009 > x86_64 x86_64 x86_64 GNU/Linux > > For 32 bit build, I got: > [exec] checking for C compiler default output file name... > [exec] configure: error: C compiler cannot create executables > [exec] See `config.log' for more details. > > BUILD FAILED > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:243: exec returned: 77 > > For 64 bit build, I got: > [exec] gcc -DHAVE_CONFIG_H -I. > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native -I./impl > -I/usr/java/default/include -I/usr/java/default/include/linux > -I/home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl > -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -m64 -MT > impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c > -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o > [exec] > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c: > In function 'Java_com_hadoop_compression_lzo_LzoCompressor_initIDs': > [exec] > > /home/rialto/kevinweil-hadoop-lzo-916aeae/src/native/impl/lzo/LzoCompressor.c:123: > error: expected expression before ',' token > [exec] make: *** [impl/lzo/LzoCompressor.lo] Error 1 > > BUILD FAILED > /home/rialto/kevinweil-hadoop-lzo-916aeae/build.xml:251: exec returned: 2 > > Has anybody seen the above ? > > Thanks > > On Mon, Jan 11, 2010 at 3:34 PM, Todd Lipcon <t...@cloudera.com> wrote: > > > Hi Ted, > > > > You need to install liblzo from EPEL: > > > > > http://fr.rpmfind.net/linux/RPM/Extras_Packages_for_Enterprise_Linux.html > > > > -Todd > > > > On Mon, Jan 11, 2010 at 3:21 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > Can someone tell me how I can install liblzo ? > > > > > > [r...@tyu-linux lzo-2.03]# uname -a > > > Linux tyu-linux 2.6.18-128.2.1.el5 #1 SMP Tue Jul 14 06:36:37 EDT 2009 > > > x86_64 x86_64 x86_64 GNU/Linux > > > [r...@tyu-linux lzo-2.03]# yum install liblzo-devel > > > Loaded plugins: fastestmirror > > > Loading mirror speeds from cached hostfile > > > * base: mirrors.usc.edu > > > * updates: mirror.san.fastserv.com > > > * addons: centos.promopeddler.com > > > * extras: mirrors.versaweb.com > > > Setting up Install Process > > > Parsing package install arguments > > > No package liblzo-devel available. > > > Nothing to do > > > > > > Thanks > > > > > > On Mon, Jan 11, 2010 at 12:45 PM, Steve Kuo <kuosen...@gmail.com> > wrote: > > > > > > > Ted, > > > > > > > > You may want to consider LZO compression, which allows splitting for > a > > > > comporessed file for Map jobs. On the other hand, gzip is not > > > splittable. > > > > > > > > Check out these links. > > > > > > > > > > > > > > > > > > http://www.cloudera.com/blog/2009/11/17/hadoop-at-twitter-part-1-splittable-lzo-compression/ > > > > http://wiki.apache.org/hadoop/UsingLzoCompression > > > > > > > > > > > > On Fri, Jan 8, 2010 at 1:13 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > > > > > > The input file is in .gz format > > > > > FYI > > > > > > > > > > On Fri, Jan 8, 2010 at 11:08 AM, Ted Yu <yuzhih...@gmail.com> > wrote: > > > > > > > > > > > My current project processes input file of size 333302161 bytes. > > > > > > What I plan to do is to split the file into equal size pieces > (and > > on > > > > > blank > > > > > > line boundary) to improve performance. > > > > > > > > > > > > I found 12 classes in 0.20.1 source code which implement > > InputSplit. > > > > > > > > > > > > If someone has written code similar to what I plan to do, please > > > share > > > > > some > > > > > > hint. > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > On Fri, Jan 8, 2010 at 2:27 AM, Amogh Vasekar < > am...@yahoo-inc.com > > > > > > > > wrote: > > > > > > > > > > > >> Hi, > > > > > >> The deprecation is due to the new evolving mapreduce ( > > > o.a.h.mapreduce > > > > ) > > > > > >> APIs. Old APIs are supported for available distributions. The > > > > equivalent > > > > > of > > > > > >> TextInputFormat is available in new API : > > > > > >> > > > > > >> > > > > > >> > > > > > > > > > > > > > > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html > > > > > >> > > > > > >> Thanks, > > > > > >> Amogh > > > > > >> > > > > > >> > > > > > >> On 1/8/10 3:47 AM, "Ted Yu" <yuzhih...@gmail.com> wrote: > > > > > >> > > > > > >> According to: > > > > > >> > > > > > >> > > > > > > > > > > > > > > > http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapred/TextInputFormat.html#isSplitable%28org.apache.hadoop.fs.FileSystem,%20org.apache.hadoop.fs.Path%29 > > > > > >> > > > > > >> isSplitable() is deprecated. > > > > > >> > > > > > >> Which method should I use to replace it ? > > > > > >> > > > > > >> Thanks > > > > > >> > > > > > >> > > > > > > > > > > > > > > > > > > > > >
