Hello Nutch Gurus,
I have spent many weeks trying to effectively fetch a large list of sites simultaneously, and have an interim solution but as it is sort of a hack I wish to ask everyone's opinion. I will try to make this as concise as possible.
My objective:
Fetch of large list of urls (several hundred) simultaneously using Nutch 2.3 with NUTCH-1947 patch applied( to get Gora 0.6 & Mongo working). Scores/ indexing is not required, only the raw page content.
Issues experienced:
- By using Nutch with a medium sized seed list (30-40 websites), I left Nutch running on an EC2 instance for about a week. At the end I was disappointed to find that it had hardly grabbed anything. Monitoring the process showed huge lag times on each loop during UpdateDB and at the beginning of Fetcher. (Up to 3hours for UpdateDB!!)
- Investigating UpdateDB showed that Nutch writes all the outlinks under each page record during the fetch stage and then during UpdateDB re-reads these and writes them back as new page records.
- According to https://wiki.apache.org/nutch/Nutch2Crawling?action="" the generate stage "Reads every url from the webtable" each time. When many sites are added to a single crawl, this makes this step take longer and longer.
Resolution method (hack):
-In config I set Fetcher.parse=true, and removed the Parse step from bin/crawl
-In FetcherReducer.java line 685, I added a for loop which takes the outLinks array and writes them to new records then calls FetchSchedule.initializeSchedule(url) on each new url. I then removed the UpdateDb step from bin/crawl
-Instead of calling multiple sites to each seed, I made an external bash script which instead initializes a new seed.txt for each website, with a single line: seedfolder/seed.txt: http://a.com, then calls generate using -crawlId http:a.com. This results in a single Mongo collection for each website http:a.com_webpage http:b.com_webpage etc. each ran in a separate instance of bin/crawl, instead of before which had a single collection 1_webpage.
So far this has proved fast, but it is somewhat of a pain to manage.
-In other words, all I do now is run generate and fetch in a loop, with fetch doing all fetching/parsing/new url saving
Thoughts:
-The current resolution method makes fetching fast. However, more management is required to run it all.
-My next course of action will be to create a command server which can launch crawl instances as needed. Before I engage on this endeavour, I would like to ask everyone's opinion.
-Usage of Mongo is due to other parts of my system which rely on it. I get the feeling that HBase seems to be the official mascot of Nutch 2, but switching to this would require a huge overhaul of code. Cassandra would be less difficult to switch to (or maybe not if the resulting tables are not CQL friendly), in any case I was not able to get it working with the current version.
Questions:
-Is the seemingly hugely inefficient reading and writing done on each loop not a big issue with HBase?
-Is multiple websites in a single crawlId (i.e. collection) actually useable in Nutch 2.x? On a single EC2 instance? How?
-Should I be looking at Hadoop clusters for simultaneous fetching of just 30-100 websites?
I would appreciate any help at all in understanding how to get the best efficiency here.
-Jon
