Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.
The following page has been changed by Arun C Murthy: http://wiki.apache.org/lucene-hadoop/NativeHadoop ------------------------------------------------------------------------------ - == Overview == + ==== Overview ==== Hadoop has native implementations of certain components for reasons of both performace & non-availability of java-implementations. This section describes the usage & details on how to build the native-libraries. - == Native Hadoop Libraries == + ==== Native Hadoop Libraries ==== - Hadoop has the following native-libraries: + Hadoop has the following native components: - * [http://www.zlib.net zlib] compression codec - Reworked zlib codec using nio's direct-buffers which gives us 60%-70% speedup. (more details [http://issues.apache.org/jira/browse/HADOOP-538 here] - * [http://www.oberhumer.com/opensource/lzo/ lzo] compression codec - Implemented due to lack of java bindings for lzo . (more details [http://issues.apache.org/jira/browse/HADOOP-851 here]) + * [http://www.zlib.net zlib] [http://lucene.apache.org/hadoop/api/org/apache/hadoop/io/compress/CompressionCodec.html compression codec] - Reworked zlib codec using nio's direct-buffers which gives us 60%-70% speedup. (more details [http://issues.apache.org/jira/browse/HADOOP-538 here]) + * [http://www.oberhumer.com/opensource/lzo/ lzo] [http://lucene.apache.org/hadoop/api/org/apache/hadoop/io/compress/CompressionCodec.html compression codec] - Implemented due to lack of java bindings for lzo . (more details [http://issues.apache.org/jira/browse/HADOOP-851 here]) - == Usage == + ==== Usage ==== It is fairly simple to use the native-hadoop libraries: * Take a look at the [#Platforms supported platforms] @@ -30, +30 @@ [[BR]] {{{INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable }}} - recheck the above steps. + If so, recheck the above steps. [[Anchor(Platforms)]] - == Supported Platforms == + ==== Supported Platforms ==== - Native-hadoop library is supported for '''*nix''' platforms only. Unfortunately it is known not to work on [http://www.cygwin.com Cygwin] and [http://www.apple.com/macosx Mac OS X] and has mainly been used on the Linux platform. + Native-hadoop library is supported for '''*nix''' platforms only. Unfortunately it is known ''not to work'' on [http://www.cygwin.com Cygwin] and [http://www.apple.com/macosx Mac OS X] and has mainly been used on the Linux platform. [http://wiki.apache.org/lucene-hadoop/HowToContribute Patches] from anyone interested in getting them working on Cygwin/MacOSX are welcome! It has been tested on the following Linux distributions: * [http://www.redhat.com/rhel/ RHEL4]/[http://fedora.redhat.com/ Fedora] * [http://www.ubuntu.com Ubuntu] * [http://www.gentoo.org Gentoo] + On all the above platforms a 32/64 bit native-hadoop library will work with a respective 32/64 bit jvm. + + [[Anchor(BuildNativeHadoop)]] + ==== Building Native Hadoop Libraries ==== + + Native-hadoop library is written in [http://en.wikipedia.org/wiki/ANSI_C ANSI C] and built using the GNU autotools-chain (autoconf, autoheader, automake, autoscan, libtool). This means it should be straight-forward to build them on any platform with a standards compliant C compiler and the GNU autotools-chain. (See [#Platforms supported platforms]) + + In particular the various packages you would need on the target platform are: + * C compiler (e.g. [http://gcc.gnu.org/ GNU C Compiler] + * GNU Autools Chain + * [http://www.gnu.org/software/autoconf/ autoconf] + * [http://www.gnu.org/software/automake/ automake] + * [http://www.gnu.org/software/libtool/ libtool] + * [http://www.zlib.net/ zlib-development package] + * [http://www.oberhumer.com/opensource/lzo/ lzo-development package] + + '''Note''': It is '''mandatory''' to have both the zlib and lzo development packages on the target platform for ''building'' the native-hadoop library; however for deployment it is sufficient to have install zlib ''or'' lzo if you wish to use only one of them. +
