Author: ab
Date: Thu Jul 8 14:13:19 2010
New Revision: 961778
URL: http://svn.apache.org/viewvc?rev=961778&view=rev
Log:
NUTCH-845 Native hadoop libs not available through maven.
Added:
nutch/trunk/lib/native/README.txt (with props)
Removed:
nutch/trunk/lib/native/Linux-amd64-64/
nutch/trunk/lib/native/Linux-i386-32/
Modified:
nutch/trunk/CHANGES.txt
nutch/trunk/build.xml
Modified: nutch/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/nutch/trunk/CHANGES.txt?rev=961778&r1=961777&r2=961778&view=diff
==============================================================================
--- nutch/trunk/CHANGES.txt (original)
+++ nutch/trunk/CHANGES.txt Thu Jul 8 14:13:19 2010
@@ -2,6 +2,8 @@ Nutch Change Log
Release 2.0 - Current Development
+* NUTCH-845 Native hadoop libs not available through maven (ab)
+
* NUTCH-843 Separate the build and runtime environments (ab)
* NUTCH-821 Use ivy in nutch builds (Enis Soztutar, jnioche)
Modified: nutch/trunk/build.xml
URL:
http://svn.apache.org/viewvc/nutch/trunk/build.xml?rev=961778&r1=961777&r2=961778&view=diff
==============================================================================
--- nutch/trunk/build.xml (original)
+++ nutch/trunk/build.xml Thu Jul 8 14:13:19 2010
@@ -159,6 +159,9 @@
<!-- local area -->
<copy file="${build.dir}/${final.name}.jar"
todir="${runtime.local}/lib"/>
+ <copy todir="${runtime.local}/lib/native">
+ <fileset dir="lib/native"/>
+ </copy>
<copy todir="${runtime.local}/conf">
<fileset dir="conf"/>
</copy>
Added: nutch/trunk/lib/native/README.txt
URL:
http://svn.apache.org/viewvc/nutch/trunk/lib/native/README.txt?rev=961778&view=auto
==============================================================================
--- nutch/trunk/lib/native/README.txt (added)
+++ nutch/trunk/lib/native/README.txt Thu Jul 8 14:13:19 2010
@@ -0,0 +1,36 @@
+This is a placeholder to put Hadoop native libraries -it's a component
+that contains platform-specific native code that significantly speeds up
+data (de)compression. Since there are no maven artifacts for this component
+the build process can't automatically download it.
+
+These libraries are purely optional, and if they are missing Hadoop will
+use corresponding pure Java components. The impact of native compression
+becomes noticeable with larger datasets and weaker CPU-s - if you notice
+that the CPU is routinely saturated when a job is sorting or reducing,
+then using these libs may help.
+
+Installation instructions
+=========================
+You can obtain the necessary files from a distribution package of Hadoop,
+e.g. hadoop-0.20.2.tar.gz. Unpack this archive, and copy the content of
+lib/native here, so that the layout looks like this:
+
+<Nutch home>/lib/native/Linux-amd64-64/...
+<Nutch home>/lib/native/Linux-i386-32/...
+
+Local runtime
+-------------
+The build process will include these native libraries when preparing
+the /runtime/local environment for running in local mode.
+
+/runtime/local/bin/nutch knows how to use these libs - if they are
+found and correctly used you should see lines like this in your logs:
+
+Distributed runtime
+-------------------
+If you want to use this component in an existing Hadoop cluster (when using
+/runtime/deploy artifacts) you need to make sure these files are placed in
+Hadoop/lib/native directory on each node, and then restart the cluster. If
+you installed the cluster from a distribution package of Hadoop then these
+libraries should already be in the right place and you shouldn't need to do
+anything else.
Propchange: nutch/trunk/lib/native/README.txt
------------------------------------------------------------------------------
svn:eol-style = native
