Dear Wiki user, You have subscribed to a wiki page or wiki category on "Lucene-hadoop Wiki" for change notification.
The following page has been changed by Arun C Murthy: http://wiki.apache.org/lucene-hadoop/NativeHadoop ------------------------------------------------------------------------------ Hadoop has native implementations of certain components for reasons of both performace & non-availability of java-implementations. This section describes the usage & details on how to build the native-libraries. - === Native Hadoop Libraries === + == Native Hadoop Libraries == + Hadoop has the following native-libraries: + * [http://www.zlib.net zlib] compression codec - Reworked zlib codec using nio's direct-buffers which gives us 60%-70% speedup. (more details [http://issues.apache.org/jira/browse/HADOOP-538 here] + * [http://www.oberhumer.com/opensource/lzo/ lzo] compression codec - Implemented due to lack of java bindings for lzo . (more details [http://issues.apache.org/jira/browse/HADOOP-851 here]) + + == Usage == + + It is fairly simple to use the native-hadoop libraries: + * Take a look at the [#Platforms supported platforms] + * Either [http://www.apache.org/dyn/closer.cgi/lucene/hadoop/ download] the prebuilt 32-bit Linux native-hadoop libraries (available as part of hadoop distribution in lib/native) or [#BuildNativeHadoop build] them yourself. + * Ensure you have either or/both of '''>zlib-1.2''' and '''>lzo2.0''' packages for your platform installed; depending on your needs. + That's it! + + The '''bin/hadoop''' script ensures that the native-hadoop library is on the ''library path'' via the system property ''-Djava.library.path='' (another alternative is to use the ''LD_LIBRARY_PATH'' variable, but the former is recommended). + + To check everything went alright check the hadoop log files for: + [[BR]] + {{{DEBUG util.NativeCodeLoader - Trying to load the custom-built native-hadoop library... + INFO util.NativeCodeLoader - Loaded the native-hadoop library}}} + + God-forbid something goes wrong, then: + [[BR]] + {{{INFO util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable + }}} + recheck the above steps. + + [[Anchor(Platforms)]] + == Supported Platforms == + + Native-hadoop library is supported for '''*nix''' platforms only. Unfortunately it is known not to work on [http://www.cygwin.com Cygwin] and [http://www.apple.com/macosx Mac OS X] and has mainly been used on the Linux platform. + + It has been tested on the following Linux distributions: + * [http://www.redhat.com/rhel/ RHEL4]/[http://fedora.redhat.com/ Fedora] + * [http://www.ubuntu.com Ubuntu] + * [http://www.gentoo.org Gentoo] +