On Fri, Jun 11, 2004 at 11:25:34PM +0200, Stefan Groschupf wrote: > > Massimo, > > >So, for its performance, and problens solved, I suggest to restore > >RequestScheduler in the Nuch CVS. > > As far as I had understand Doug, there was several reasons to delete > the request scheduler. > One of the major points was that there was 2 fetcher implementation and > that make no sense to maintain 2 implementation in one project. > The choice to delete the Request-Scheduler was done from my > understanding since it was the implementation with no unit tests and a > set of people was reporting problems with it. > Feel free to request feature you miss in the actual implementation and > or patches are mostly welcome. > > As already mentioned I will track down the fetcher performance problem > but since this is my spare time just give me a bit!!!
A while ago, I read through RequestScheduler source once. Though not being able to grasp its author's thought completely, I can clearly see that it has a rather sophisticated implementation. Quite an effort by its author or authors. Furthermore, I have made many runs with it, long or short, but never had problems that I can not attribute to other casues. It used to hang in my smaller boxes (~ 1GB in memory), only a few times. But no problem since being moved to newer ones with 4GB memory. So far, all the reported problems seem to be in lengthy runs (without clear spec about hardwares used). And there have been no apple-to-apple comparison between Fetcher.java and RequestScheduler.java. Probably it is premature to delete it from source tree now. I would like to have it kept, at least for a while, even though no one has the time or energy to make it up to date. It does give you very good performance. It seems to me that the reported problems may have more to do with protocol client(s) or malfunctioning sites/pages (intentionally or not), or hardware resources. Also can't say for sure if Fetcher.java won't have similar problem. Massimo might be able to help us by doing a comparion? Just my two cents. John ------------------------------------------------------- This SF.Net email is sponsored by the new InstallShield X. >From Windows to Linux, servers to mobile, InstallShield X is the one installation-authoring solution that does it all. Learn more and evaluate today! http://www.installshield.com/Dev2Dev/0504 _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
