Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "NutchHadoopTutorial" page has been changed by AlexMc. http://wiki.apache.org/nutch/NutchHadoopTutorial?action=diff&rev1=19&rev2=20 -------------------------------------------------- - = How to Setup Nutch (V1.0) and Hadoop = + = How to Setup Nutch (V1.1) and Hadoop = -------------------------------------------------------------------------------- - After searching the web and mailing lists, it seems that there is very little information on how to setup Nutch using the Hadoop (formerly NDFS) distributed file system (HDFS) and MapReduce. The purpose of this tutorial is to provide a step-by-step method to get Nutch running with Hadoop file system on multiple machines, including being able to both index (crawl) and search across multiple machines. + + Note: Originally this (NutchHadoopTutorial0.8) was written for version 0.8 of Nutch. This has been edited by people other than the original author so statements like "I did this" or "I recommend that" are slightly misleading. + + -------------------------------------------------------------------------------- + + By default, out of the box, Nutch runs in a single process on one machine. This may suit you fine if you have a small site to crawl and index, but most people choose Nutch because of its capability to run on a Hadoop cluster. This gives you the benefit of a distributed file system (HDFS) and MapReduce processing style. The purpose of this tutorial is to provide a step-by-step method to get Nutch running with Hadoop file system on multiple machines, including being able to both index (crawl) and search across multiple machines. This document does not go into the Nutch or Hadoop architecture. It only tells how to get the systems up and running. At the end of the tutorial though I will point you to relevant resources if you want to know more about the architecture of Nutch and Hadoop. + The tutorial comes in two phases. Firstly we get Hadoop running on a single machine (a bit of a simple cluster!) and then more than one machine. + Some things are assumed for this tutorial: - First, I performed some setup and using root level access. This included setting up the same user across multiple machines and setting up a local filesystem outside of the user's home directory. Root access is not required to setup Nutch and Hadoop (although sometimes it is convienent). If you do not have root access, you will need the same user setup across all machines which you are using and you will probably need to use a local filesystem inside of your home directory. + First: I performed some setup and using root level access. This included setting up the same user across multiple machines and setting up a local filesystem outside of the user's home directory. Root access is not required to setup Nutch and Hadoop (although sometimes it is convenient). If you do not have root access, you will need the same user setup across all machines which you are using and you will probably need to use a local filesystem inside of your home directory. - Two, all boxes will need an SSH server running (not just a client) as Hadoop uses SSH to start slave servers. + Two: all boxes will need an SSH server running (not just a client) as Hadoop uses SSH to start slave servers. Although we try to explain how to set up ssh so that communication between machines does not require a password you may need to learn how to do that elsewhere. - Three, this tutorial uses Whitebox Enterprise Linux 3 Respin 2 (WHEL). For those of you who don't know Whitebox, it is a RedHat Enterprise Linux clone. You should be able to follow along for any linux system, but the systems I use are Whitebox. + Three: This tutorial uses Whitebox Enterprise Linux 3 Respin 2 (WHEL). For those of you who don't know Whitebox, it is a RedHat Enterprise Linux clone. You should be able to follow along for any linux system, but the systems I use are Whitebox. (Later versions of this document have been tested using Ubuntu Linux, but as before - Four, this tutorial uses Nutch 0.8 Dev Revision 385702, and may not be compatible with future releases of either Nutch or Hadoop. + Four: This tutorial was originally written for Nutch 0.8 Dev Revision 385702, but has been updated to work with Nutch 1.1RC. It may not be compatible with future releases of either Nutch or Hadoop. - (AlexMc is trying to update this article to make it consistent with Nutch version 1.0 ) - Five, for this tutorial we setup nutch across 6 different computers. If you are using a different number of machines you should still be fine but you should have at least two different machines to prove the distributed capabilities of both HDFS and MapReduce. + Five: For this tutorial we setup nutch across 6 different computers. If you are using a different number of machines you should still be fine but you should have at least two different machines to prove the distributed capabilities of both HDFS and MapReduce. - Six, in this tutorial we build Nutch from source. There are nightly builds of both Nutch and Hadoop available and I will give you those urls later. + Six: Remember that this is a tutorial from my personal experience setting up Nutch and Hadoop. If something doesn't work for you try searching and sending a message to the Nutch or Hadoop users mailing list. Suggestions or tips are welcome. Why not add them to the end of this Wiki page? - Seven, remember that this is a tutorial from my personal experience seting up Nutch and Hadoop. If something doesn't work for you try searching and sending a message to the Nutch or Hadoop users mailing list. And as always suggestions are welcome to help improve this tutorial for others. + Seven: We assume that you are a Java programmer familiar with the concepts of JAVA_HOME, ant build tool, subversion, IDEs and such like. == Our Network Setup == -------------------------------------------------------------------------------- + - First let me layout the computers that we used in our setup. To setup Nutch and Hadoop we had 7 commodity computers ranging from 750Mghz to 1.0 Ghz. Each computer had at least 128 Megs of RAM and at least a 10 Gigabyte hard drive. One computer had dual 750 Mghz CPUs and another had dual 30 Gigabyte hard drives. All of these computers were purchasedfor under $500.00 at a liquidation sale. I am telling you this to let you know that you don't have to have big hardware to get up and running with Nutch and Hadoop. Our computers were named like this: + First let me layout the computers that we used in our setup. To setup Nutch and Hadoop we had 7 commodity computers ranging from 750Mghz to 1.0 Ghz. Each computer had at least 128 Megs of RAM and at least a 10 Gigabyte hard drive. One computer had dual 750 Mghz CPUs and another had dual 30 Gigabyte hard drives. All of these computers were purchased for under $500.00 at a liquidation sale. I am telling you this to let you know that you don't have to have big hardware to get up and running with Nutch and Hadoop. Our computers were named like this: {{{ devcluster01 @@ -40, +47 @@ == Downloading Nutch and Hadoop == -------------------------------------------------------------------------------- - Both Nutch and Hadoop are downloadable from the apache website. The necessary Hadoop files are bundled with Nutch so unless you are going to be developing Hadoop you only need to download Nutch. + Both Nutch and Hadoop are downloadable from the Apache website. The necessary Hadoop files are bundled with Nutch so unless you are going to be developing Hadoop you only need to download Nutch. We built Nutch from source after downloading it from its subversion repository. - There are nightly builds of both Nutch and Hadoop here: + Nightly builds of Nutch can be found here: http://hudson.zones.apache.org/hudson/job/Nutch-trunk/ - http://cvs.apache.org/dist/lucene/hadoop/nightly/ + At time of writing this version (Jun 2010) Nutch includes Hadoop Jars version 0.20.2 - I am using eclipse for development so I used the eclipse plugin for subversion to download both the Nutch and Hadoop repositories. The subversion plugin for eclipse can be downloaded through the update manager using the url: + You can get a packaged tarball or extract from subversion. Knowing how to use tar or subversion is outside of the scope of this tutorial. Once you have a subversion client you can either browse the Nutch subversion webpage at: - http://subclipse.tigris.org/update_1.0.x - - If you are not using eclipse you will need to get a subversion client. Once you have a subversion client you can either browse the Nutch subversion webpage at: - - http://lucene.apache.org/nutch/version_control.html + http://nutch.apache.org/version_control.html Or you can access the Nutch subversion repository through the client at: - http://svn.apache.org/repos/asf/lucene/nutch/ + http://svn.apache.org/repos/asf/nutch/ (previously at http://svn.apache.org/repos/asf/lucene/nutch/ when Nutch was a part of Lucene) - I checked out the main trunk into my eclipse but it can be checked out to a standard filesystem as well. We are going to use ant to build it so if you have java and ant installed you should be fine. + We are going to use ant to build it so if you have java and ant installed you should be fine. I am not going to go into how to install java or ant, if you are working with this level of software you should know how to do that and there are plenty of tutorial on building software with ant. If you want a complete reference for ant pick up Erik Hatcher's book "''Java Development with Ant''": http://www.manning.com/hatcher + It is worth noting that previous versions of Nutch came already built. But nowadays the release is just source code and so does have to be built before use. == Building Nutch and Hadoop == -------------------------------------------------------------------------------- - Once you have Nutch downloaded go to the download directory where you should see the following folders and files: + Once you have Nutch downloaded and unpacked look inside it where you should see the following folders and files: {{{ + bin @@ -124, +128 @@ /tomcat (only on one server for searching) }}} - I am not going to go into detail about how to install tomcat as again there are plenty of tutorials on how to do that. I will say that we removed all of the wars from the webapps directory and created a folder called ROOT under webapps into which we unzipped the Nutch war file (nutch-0.8-dev.war). This makes it easy to edit configuration files inside of the Nutch war + I am not going to go into detail about how to install Tomcat as again there are plenty of tutorials on how to do that. I will say that we removed all of the wars from the webapps directory and created a folder called ROOT under webapps into which we unzipped the Nutch war file (nutch-0.8-dev.war). This makes it easy to edit configuration files inside of the Nutch war So log into the master nodes and all of the slave nodes as root. Create the nutch user and the different filesystems with the following commands: @@ -187, +191 @@ cp id_rsa.pub authorized_keys }}} - You only have to run the ssh-keygen on the master node. On each of the slave nodes after the filesystem is created you will just need to copy the keys over using scp. + You only have to run the ssh-keygen on the master node. On each of the slave nodes after the filesystem is created you will just need to copy the keys over using scp. eg to send the authorisation from to devcluster02 we might do this on devcluster01 {{{ scp /nutch/home/.ssh/authorized_keys nu...@devcluster02:/nutch/home/.ssh/authorized_keys }}} - You will have to enter the password for the nutch user the first time. An ssh propmt will appear the first time you login to each computer asking if you want to add the computer to the known hosts. Answer yes to the propmt. Once the key is copied you shouldn't have to enter a password when logging in as the nutch user. Test it by logging into the slave nodes that you just copied the keys to: + You will have to enter the password for the nutch user the first time. An ssh prompt will appear the first time you login to each computer asking if you want to add the computer to the known hosts. Answer yes to the prompt. Once the key is copied you shouldn't have to enter a password when logging in as the nutch user. Test it by logging into the slave nodes that you just copied the keys to: {{{ ssh devcluster02 @@ -202, +206 @@ }}} Once we have the ssh keys created we are ready to start deploying nutch to all of the slave nodes. + + (Note: this is a rather simple example of how to set up ssh without requiring a passphrase. There are other documents available which can help you with this if you have problems. It is important to test that the nutch user can ssh to all of the machines in your cluster so don't skip this stage) == Deploy Nutch to Single Machine ==
