Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "NutchHadoopTutorial" page has been changed by AlexMc. http://wiki.apache.org/nutch/NutchHadoopTutorial?action=diff&rev1=20&rev2=21 -------------------------------------------------- localhost }}} - It comes this way to start so you shouldn't have to make any changes. Later we will add all of the nodes to this file, one node per line. Below is an example hadoop-site.xml file. [TO BE FIXED] + It comes this way to start so you shouldn't have to make any changes. Later we will add all of the nodes to this file, one node per line. + + Previously all the Hadoop configuration was in one file (hadoop-site.xml) but now we need to put roughly the same data in separate files. See http://hadoop.apache.org/common/docs/current/quickstart.html for more information. We are basically adding property entries inside the configuration tags... + + conf/core-site.xml: {{{ <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> - <!-- Put site-specific property overrides in this file. --> - <configuration> + <property> + <name>fs.default.name</name> + <value>hdfs://devcluster01:9000</value> + <description> + Where to find the Hadoop Filesystem through the network. + Note 9000 is not the default port. + (This is slightly changed from previous versions which didnt have "hdfs") + </description> + </property> + </configuration> + }}} + + The fs.default.name property is used by nutch to determine the filesystem that it is going to use. Since we are using the hadoop filesystem we have to point this to the hadoop master or name node. In this case it is hdfs://devcluster01:9000 which is the server that houses the name node on our network. + + The hadoop package really comes with two components. One is the distributed filesystem. Two is the mapreduce functionality. While the distibuted filesystem allows you to store and replicate files over many commodity machines, the mapreduce package allows you to easily perform parallel programming tasks. + + + + conf/hdfs-site.xml: + + {{{ + <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> + <!-- Put site-specific property overrides in this file. --> + <configuration> <property> - <name>fs.default.name</name> + <name>dfs.name.dir</name> + <value>/nutch/filesystem/name</value> - <value>devcluster01:9000</value> - <description> - The name of the default file system. Either the literal string - "local" or a host:port for NDFS. - </description> </property> + + <property> + <name>dfs.data.dir</name> + <value>/nutch/filesystem/data</value> + </property> + + <property> + <name>dfs.replication</name> + <value>1</value> + </property> + + </configuration> + }}} + + + conf/mapred-site.xml: + + {{{ + <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> + <!-- Put site-specific property overrides in this file. --> + <configuration> <property> <name>mapred.job.tracker</name> @@ -287, +330 @@ The host and port that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. + Note 9001 is not the default port. </description> </property> @@ -307, +351 @@ </property> <property> - <name>dfs.name.dir</name> - <value>/nutch/filesystem/name</value> - </property> - - <property> - <name>dfs.data.dir</name> - <value>/nutch/filesystem/data</value> - </property> - - <property> <name>mapred.system.dir</name> <value>/nutch/filesystem/mapreduce/system</value> </property> @@ -326, +360 @@ <value>/nutch/filesystem/mapreduce/local</value> </property> - <property> - <name>dfs.replication</name> - <value>1</value> - </property> - </configuration> }}} - The fs.default.name property is used by nutch to determine the filesystem that it is going to use. Since we are using the hadoop filesystem we have to point this to the hadoop master or name node. In this case it is devcluster01:9000 which is the server that houses the name node on our network. - - The hadoop package really comes with two components. One is the distributed filesystem. Two is the mapreduce functionality. While the distibuted filesystem allows you to store and replicate files over many commodity machines, the mapreduce package allows you to easily perform parallel programming tasks. The distributed file system has name nodes and data nodes. When a client wants to manipulate a file in the file system it contacts the name node which then tells it which data node to contact to get the file. The name node is the coordinator and stores what blocks (not really files but you can think of them as such for now) are on what computers and what needs to be replicated to different data nodes. The data nodes are just the workhorses. They store the actual files, serve them up on request, etc. So if you are running a name node and a data node on the same computer it is still communicating over sockets as if the data node was on a different computer. @@ -363, +389 @@ {{{ bin/hadoop namenode -format }}} + + And check the logs directory looking for errors. Now that we have our hadoop configured and our slaves file configured it is time to start up hadoop on a single node and test that it is working properly. To start up all of the hadoop servers on the local machine (name node, data node, mapreduce tracker, job tracker) use the following command as the nutch user: @@ -379, +407 @@ If everything has been setup correctly you should see output saying that the name node, data node, job tracker, and task tracker services have started. If this happens then we are ready to test out the filesystem. You can also take a look at the log files under /nutch/search/logs to see output from the different daemons services we just started. + You might want to look at http://localhost:50070/ with a web browser to confirm that the NameNode is up and running. (Replace localhost with devcluster01 or whatever you main host is called) + + You can also look at http://localhost:50030/ to confirm that the JobTracker is up and running. (These ports seem to remain the same no matter that we entered "9000" and "9001" above. + + + + To test the filesystem we are going to create a list of urls that we are going to use later for the crawl. Run the following commands: {{{ cd /nutch/search - mkdir urls + mkdir urlsdir - vi urls/urllist.txt + vi urlsdir/urllist.txt http://lucene.apache.org }}} @@ -393, +428 @@ {{{ cd /nutch/search - bin/hadoop dfs -put urls urls + bin/hadoop dfs -put urlsdir urlsdir }}} You should see output stating that the directory was added to the filesystem. You can also confirm that the directory was added by using the ls command: @@ -419, +454 @@ scp -r /nutch/search/* nu...@computer:/nutch/search }}} - Do this for every computer you want to use as a slave node. Then edit the slaves file, adding each slave node name to the file, one per line. You will also want to edit the hadoop-site.xml file and change the values for the map and reduce task numbers, making this a multiple of the number of machines you have. For our system which has 6 data nodes I put in 32 as the number of tasks. The replication property can also be changed at this time. A good starting value si something like 2 or 3. *(see Note at bottom about possibly having to clear filesystem of new datanodes). Once this is done you should be able to startup all of the nodes. + Do this for every computer you want to use as a slave node. Then edit the slaves file, adding each slave node name to the file, one per line. You will also want to edit the hadoop-site.xml file and change the values for the map and reduce task numbers, making this a multiple of the number of machines you have. For our system which has 6 data nodes I put in 32 as the number of tasks. The replication property can also be changed at this time. A good starting value is something like 2 or 3. *(see Note at bottom about possibly having to clear filesystem of new datanodes). Once this is done you should be able to startup all of the nodes. To start all of the nodes we use the exact same command as before: @@ -451, +486 @@ {{{ cd /nutch/search - bin/nutch crawl urls -dir crawled -depth 3 + bin/nutch crawl urlsdir -dir crawled -depth 3 }}} - We are using the nutch crawl command. The urls is the urls directory that we added to the distributed filesystem. The -dir crawled is the output directory. This will also go to the distributed filesystem. The depth is 3 meaning it will only get 3 page links deep. There are other options you can specify, see the command documentation for those options. + We are using the nutch crawl command. The urlsdir is the urls directory that we added to the distributed filesystem. (I've called it "urlsdir" to make it clearer that it isn't merely the *file* containing urls). The -dir crawled is the output directory. This will also go to the distributed filesystem. The depth is 3 meaning it will only get 3 page links deep. There are other options you can specify, see the command documentation for those options. You should see the crawl startup and see output for jobs running and map and reduce percentages. You can keep track of the jobs by pointing you browser to the master name node: + + http://devcluster01:50070 + + and Mapreduce administration at http://devcluster01:50030 @@ -621, +660 @@ == Resources == -------------------------------------------------------------------------------- + Hadoop Quickstart: + http://hadoop.apache.org/common/docs/current/quickstart.html + Google MapReduce Paper: If you want to understand more about the MapReduce architecture used by Hadoop it is useful to read about the Google implementation.
