Some list of frequenty updated pages will be crawled weekly (depth -1), and put result to the segment directory (~ 1000 page).
Chirag Chaman wrotte:
One of the more important questions is How frequently is the data updated?
The answer to this tells you how to configure you DB/fetch/crawl servers.
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Thursday, April 14, 2005 12:08 PM
To: [email protected]
Subject: Re: Nutch - new public server
Dear Users,
I would like make a new search server with the following: - avarage 10 queries per sec - ~ 30 Million pages
How to make it? I think there are need: - last stable version of nutch (0.6). - 2-3 search server (P4 1-2 GByte RAM 2 network adapter) - One Opteron db server, with 30 GByte RAM, 700 GByte HDD (350 GByte for search db and backup, and 350 GByte for actual crawl) and 2-4 network adapter (one for crawl, others to search servers), Sun 1.5 64bit java It's good idea?
Thanks, Ferenc
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Nutch-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-general
