Thanks for the quick response. Dennis, I'm not sure how to change the setting in the NutchBean, however I set the variable int hitsPerSite in search.jsp instead.
On a performance note, do you recommend loading the indexes directory in ram (tmpfs on linux) to reduce IO and increase performance? I guess it depends on how large the index is and how much ram is available, however it sounds like a too good to be true method of squeezing out extra performance from a nutch web server. Your thoughts pls. Regards, Hilkiah G. Lavinier MEng (Hons), ACGI 6 Winston Lane, Goodwill, Roseau, Dominica Mbl: (767) 275 3382 Hm : (767) 440 3924 Fax: (767) 440 4991 VoIP USA: (646) 432 4487 Email: [EMAIL PROTECTED] Email: [EMAIL PROTECTED] IM: Yahoo hilkiah / MSN [EMAIL PROTECTED] IM: ICQ #8978201 / AOL hilkiah21 ----- Original Message ---- From: Dennis Kubes <[EMAIL PROTECTED]> To: [email protected] Sent: Saturday, January 19, 2008 7:24:03 PM Subject: Re: distributed search servers Hilkiah Lavinier wrote: > Hi all, > > Have a distributed search issue I need some advice on. The scenario is that I have tomcat running off one server and two nutch search servers running off two other machines (so 3 machines in total). I've setup the nutch war to correctly call the search servers and they respond. Problem is I get duplicate results. Now I have the same data/information from the crawl copied on both machines so the crawl data is replicated on both machines. > > Questions: > 1) how do I prevent the duplicate response? If I start a third search server I only get two duplicate responses so it doesn't seem to increase with the number of search servers In your query or in NutchBean set the hitsPerSite=1, here is an example: Duplicates: http://search.isc.swlabs.org/search.jsp?lang=en&query=java No Duplicates: http://search.isc.swlabs.org/search.jsp?lang=en&query=java&hitsPerSite=1 This is based on hostname so for instance java.net and www.java.net will be considered different even though they are the same. The latter problem has not been corrected yet in Nutch, but we are working on it. > 2) does tomcat wait for ALL search servers to respond before displaying the query result or does it display the result as soon as one server responds? Yes, to a timeout value. If one goes down it will slow down the entire search cluster. > 3) in terms of load sharing, what is the best approach for distributed search servers? If you are looking at a round-robin sort of load balancing I would say two nutch servers hitting different search servers with replicated content fronted by an apache server or hardware load balancer. Remember that the entire search can still be up even if one or more search servers fail. I would worry more about clustering the front end search website than load balancing the search servers but it all depends on what your goal is. For a www search we don't care if a few of the search servers are down as long as the search is functional. Dennis Kubes > > Any help would be greatly appreciated! > > Thanks, > > Hilkiah G. Lavinier MEng (Hons), ACGI > 6 Winston Lane, > Goodwill, > Roseau, Dominica > Mbl: (767) 275 3382 > Hm : (767) 440 3924 > Fax: (767) 440 4991 > VoIP USA: (646) 432 4487 > > Email: [EMAIL PROTECTED] > Email: [EMAIL PROTECTED] > IM: Yahoo hilkiah / MSN [EMAIL PROTECTED] > IM: ICQ #8978201 / AOL hilkiah21 > > > > > > ____________________________________________________________________________________ > Looking for last minute shopping deals? > Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs
