Hi, I just realized that my nutch didn't crawl the articles/pages (depth 2) which shown on frontpage. My target URL is: http://bappenas.go.id
As shown on that frontpage (top right below the slider banners) three is a text link: "Kerjasama Pembangunan Indonesia-Australia Setelah PM Tony Abbot" and its URL: http://bappenas.go.id/berita-dan-siaran-pers/kerjasama-pembangunan-indonesia-australia-setelah-pm-tony-abbot/?&kid=1390691937 I tried to search with keyword "Liberal Party" (with quotes) which appear on link (page) above but has no result :( Following is the search link queried: http://bappenas.go.id/index.php/bappenas_search/result?q=%22Liberal+Party%22 I use individual script to crawl below: === # Defines env variables export JAVA_HOME="/opt/searchengine/jdk1.7.0_45" export PATH="$JAVA_HOME/bin:$PATH" NUTCH="/opt/searchengine/nutch" # Start by injecting the seed url(s) to the nutch crawldb: $NUTCH/bin/nutch inject $NUTCH/BappenasCrawl/crawldb $NUTCH/urls/seed.txt # Generate fetch list $NUTCH/bin/nutch generate $NUTCH/BappenasCrawl/crawldb $NUTCH/BappenasCrawl/segments # last segment export SEGMENT=$NUTCH/BappenasCrawl/segments/`ls -tr $NUTCH/BappenasCrawl/segments|tail -1` # Launch the crawler! $NUTCH/bin/nutch fetch $SEGMENT -noParsing # Parse the fetched content: $NUTCH/bin/nutch parse $SEGMENT # We need to update the crawl database to ensure that for all future crawls, Nutch only checks the already crawled pages, and only fetches new and changed pages. $NUTCH/bin/nutch updatedb $NUTCH/BappenasCrawl/crawldb $SEGMENT -filter -normalize # Indexing our crawl DB with solr $NUTCH/bin/nutch solrindex http://localhost:8080/solr/bappenasgoid/$NUTCH/BappenasCrawl/crawldb -dir $NUTCH/BappenasCrawl/segments === I run this script daily but it looks it never reach the single article pages which shown on the frontpage. If I read Tejas explained on another thread (shown below), should I two or three times loops (generate -> fetch -> parse -> update) to produce 2 or 3 depth levels? QUOTES from Tejas' e-mail (subject: Questions/issues with nutch): ***************** On Sat, Jun 29, 2013 at 2:49 PM, Tejas Patil <[email protected]>wrote: Yes. Nutch would parse the HTML and extract the content out of it. Tweaking around the code surrounding the parser would have made that happen. If you did something else, would you mind sharing it ? The "depth" is used by the Crawl class in 1.x which is deprecated in 2.x. Use bin/crawl instead. While running the "bin/crawl" script, the "<numberOfRounds>" option is nothing but the depth till which you want the crawling to be performed. If you want to use the individual commands instead, run generate -> fetch -> parse -> update multiple times. The crawl script internally does the same thing. eg. If you want to fetch till depth 3, this is how you could do: inject -> (generate -> fetch -> parse -> update) -> (generate -> fetch -> parse -> update) -> (generate -> fetch -> parse -> update) -> solrindex ***************** I also has commented line below on regex-urlfilter.txt file: # skip URLs containing certain characters as probable queries, etc. #-[?*!@=] Apps: nutch 1.7 and Solr 4.5.1 Thank you so much! -- wassalam, [bayu]
