Hello Stefan,
I have already written a component that implements this round robin searching functionality some time ago - but right now it is not working correctly with latest nutch SVN code - anyway I have plans to update it. It was done inside modified NutchBean - it was selecting the group of servers to be used for particular request in round-robin fashion and in case of failure it was moving the current server group to inactive pool and retrying using another group.
There was a separate recovery thread that was checking from time to time
if inactive pool contains some groups and was trying to recover it.
So in addition to load-balancing all requests among cluster of search server groups it was also providing fault tolerance (automatic detection of inactive nodes with recovery).
Our plan was to use two or more tomcat servers with NutchBean each configured to use all search server groups. This will remove single point of failure during search.
So if there is enough interest for it I can downgrade it to JDK 1.4 (I am using java.util.concurrent) and send it as a patch.
Regards, Piotr
Stefan Groschupf wrote:
I notice similar behaviors.
I guess the backend servers does not answering fast enough.
I was thinking about to have multiple search server groups that have identical content and then query groups in a round robbing style.
What people think about this idea?
It is already easy to setup multiple tomcat that use different search servers and simply split traffic by adding 2 or n ip to your dns for the same domain.
Stefan
Am 18.05.2005 um 16:59 schrieb [EMAIL PROTECTED]:
Dear Users!
Firstly sorry my bad English.
I read Stephans great documentation at http://wiki.media-style.com/ display/nutchDocu/.
I maked a frontend (P4 3 GByte RAM, Tomcat 5.5.7 java 1.4.08) with 3 backend with 12 million pages ( 4million / backend AMD64 4 GByte RAM 64 bit linux with jdk 1.5_03).
When I start using it with 3-5 queries / sec, after 1-2 minute the frontend does'nt answer to the requests.
In the Tomcat manager / status I see there is many thread busy (150 and it increasses, now 241), and these are with Stage 'S' (Service).
The backend with usage: top 40-60 % CPU. The frontend with usage: 5% CPU.
Have you any idea what is the problem?
Best Regards, Ferenc
--------------------------------------------------------------- company: http://www.media-style.com forum: http://www.text-mining.org blog: http://www.find23.net
