In that case a central DB server would be okay, and you can move the segments around using rsync to the search servers. I've found this to be a more "hassle-free" solution that using NDFS, but I'm certain given the new developments the latter will be better in the coming months.
I would STRONGLY recommend getting a 4-disk RAID setup for the WebDB. That is your main bottleneck. 1-2 NICS is fine So here's a look at the setup 1. "Souped-up" DB server - Dual CPU, 4 GB Ram (min) RAID 5 or 10, 1-2 NICS 2. Basic Search Servers - Single/Dual CPU, Maximum RAM, Single IDE/SATA drive (or 2 for redundancy) 3. Basic Web Servers - Single/Dual CPU, Medium RAM -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] Sent: Friday, April 15, 2005 4:54 AM To: [EMAIL PROTECTED] Subject: Re: [Nutch-general] RE: Nutch - new public server I would like update most of pages monthly. Some list of frequenty updated pages will be crawled weekly (depth -1), and put result to the segment directory (~ 1000 page). Chirag Chaman wrotte: >One of the more important questions is >How frequently is the data updated? > >The answer to this tells you how to configure you DB/fetch/crawl servers. > > >-----Original Message----- >From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] >Sent: Thursday, April 14, 2005 12:08 PM >To: [email protected] >Subject: Re: Nutch - new public server > >Dear Users, > >I would like make a new search server with the following: >- avarage 10 queries per sec >- ~ 30 Million pages > >How to make it? >I think there are need: >- last stable version of nutch (0.6). >- 2-3 search server (P4 1-2 GByte RAM 2 network adapter) >- One Opteron db server, with 30 GByte RAM, 700 GByte HDD (350 GByte >for search db and backup, and 350 GByte for actual crawl) and 2-4 >network adapter (one for crawl, others to search servers), Sun 1.5 >64bit java It's good idea? > >Thanks, Ferenc > > > > >------------------------------------------------------- >SF email is sponsored by - The IT Product Guide Read honest & candid >reviews on hundreds of IT Products from real users. >Discover which products truly live up to the hype. Start reading now. >http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click >_______________________________________________ >Nutch-general mailing list >[email protected] >https://lists.sourceforge.net/lists/listinfo/nutch-general > > > >
