I have downloaded, installed, and successfully played around with nutch and have to say I am quite impressed with the power of this program. Basically, I would like to hire a nutch expert to help me layout a plan on how to use nutch for the following scenario. We have about 1000 domains that we need to "deep crawl" regularly. Our original estimates are an overall url database of 50 - 100 million urls. We have a bit of computing horsepower with which to distribute the crawling and searching. (Most of which are dual xeons with a good amount of ram :)) I am guessing that we need to have a main index server with a few terabytes of storage. Anyways, additional details can be given on reply. I would be willing to pay someone $500 to help us setup such a system..um..the right way. I have a feeling that on my own there will be a lot of trial and error that I would like to avoid. If anyone is interested let me know. Thanks, Michael
------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
