Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The "NutchHadoopTutorial" page has been changed by AlexMc. http://wiki.apache.org/nutch/NutchHadoopTutorial?action=diff&rev1=23&rev2=24 -------------------------------------------------- {{{ cd /nutch/search - bin/nutch crawl urlsdir -dir crawled -depth 3 + bin/nutch crawl urlsdir -dir crawl -depth 3 }}} - We are using the nutch crawl command. The urlsdir is the urls directory that we added to the distributed filesystem. (I've called it "urlsdir" to make it clearer that it isn't merely the *file* containing urls). The -dir crawled is the output directory. This will also go to the distributed filesystem. The depth is 3 meaning it will only get 3 page links deep. There are other options you can specify, see the command documentation for those options. + We are using the nutch crawl command. The urlsdir is the urls directory that we added to the distributed filesystem. (I've called it "urlsdir" to make it clearer that it isn't merely the *file* containing urls). The "-dir crawl" is the output directory. This will also go to the distributed filesystem. The depth is 3 meaning it will only get 3 page links deep. There are other options you can specify, see the command documentation for those options. You should see the crawl startup and see output for jobs running and map and reduce percentages. You can keep track of the jobs by pointing you browser to the master name node: @@ -505, +505 @@ You can also startup new terminals into the slave machine and tail the log files to see detailed output for that slave node. The crawl will probably take a while to complete. When it is done we are ready to do the search. + == Testing the Crawl == + + You might want to try some of these commands before doing a search + + {{{ + bin/nutch readlinkdb crawl/linkdb -dump /tmp/linksdir + mkdir /nutch/search/output/ + bin/hadoop dfs -copyToLocal /tmp/linksdir /nutch/search/output/linksdir + less /nutch/search/output/linksdir/* + }}} + + Or if we want to look at the whole thing as a text file we might try + + {{{ + bin/nutch readdb crawl/crawldb -dump /tmp/entiredump + bin/hadoop dfs -copyToLocal /tmp/entiredump /nutch/search/output/entiredump + less /nutch/search/output/entiredump/* + }}} == Performing a Search == -------------------------------------------------------------------------------- @@ -515, +533 @@ Once the index has been created on the DFS you can use the hadoop copyToLocal command to move it to the local file system as such. {{{ - bin/hadoop dfs -copyToLocal crawled /d01/local/ + bin/hadoop dfs -copyToLocal crawl /d01/local/ }}} Your crawl directory should have an index directory which should contain the actual index files. Later when working with Nutch and Hadoop if you have an indexes directory with folders such as part-xxxxx inside of it you can use the nutch merge command to merge segment indexes into a single index. The search website when pointed to local will look for a directory in which there is an index folder that contains merged index files or an indexes folder that contains segment indexes. This can be a tricky part because your search website can be working properly but if it doesn't find the indexes, all searches will return nothing. @@ -542, +560 @@ <property> <name>searcher.dir</name> - <value>/d01/local/crawled</value> + <value>/d01/local/crawl</value> </property> </configuration> @@ -550, +568 @@ The fs.default.name property is now pointed locally for searching the local index. Understand that at this point we are not using the DFS or MapReduce to do the searching, all of it is on a local machine. - The searcher.dir directory is the directory where the index and resulting database are stored on the local filesystem. In our crawl command earlier we used the crawled directory which stored the results in crawled on the DFS. Then we copied the crawled folder to our /d01/local directory on the local fileystem. So here we point this property to /d01/local/crawled. The directory which it points to should contain not just the index directory but also the linkdb, segments, etc. All of these different databases are used by the search. This is why we copied over the crawled directory and not just the index directory. + The searcher.dir directory is the directory where the index and resulting database are stored on the local filesystem. In our crawl command earlier we used the crawl directory which stored the results in "crawl" on the HDFS. Then we copied the crawl folder to our /d01/local directory on the local fileystem. So here we point this property to /d01/local/crawl. The directory which it points to should contain not just the index directory but also the linkdb, segments, etc. All of these different databases are used by the search. This is why we copied over the entire crawl directory and not just the index directory. + Once the nutch-site.xml file is edited then the application should be ready to go. You can start tomcat with the following command: @@ -593, +612 @@ On each of the search servers you would use the startup the distributed search server by using the nutch server command like this: {{{ - bin/nutch server 1234 /d01/local/crawled + bin/nutch server 1234 /d01/local/crawl }}} The arguments are the port to start the server on which must correspond with what you put into the search-servers.txt file and the local directory that is the parent of the index folder. Once the distributed search servers are started on each machine you can startup the website. Searching should then happen normally with the exception of search results being pulled from the distributed search server indexes. In the logs on the search website (usually catalina.out file), you should see messages telling you the number of servers and segments the website is attached to and searching. This will allow you to know if you have your setup correct.
