Thanks for the info. I was thinking of setting it up on a 100 machine cluster and checking out how it performs. Based on the number of questions I see here regarding trivial sized fetches (<100M pages a day) it looks like this will not work for us.
Regards, Dejan -----Original Message----- From: Karol Rybak [mailto:[EMAIL PROTECTED] Sent: Friday, January 04, 2008 4:16 AM To: [email protected] Subject: Re: Nutch - crashed during a large fetch, how to restart? Hi there i had similiar experiences with nutch - general unstability, i experienced a vast range of different crashes and hangups. Actually it worked stable only on single machine, however when number of urls grown higher it became useless. So generally speaking i would not recommend it for using in production environment to anyone. Maybe there is a way to set it up properly for large internet fetches, but i couldn't accomplish that. -- Karol Rybak Programista / Programmer Sekcja aplikacji / Applications section Wyższa Szkoła Informatyki i Zarządzania / University of Internet Technology and Management +48(17)8661277
