Have you tried to change the gc options?
On 8/9/07, purpureleaf <[EMAIL PROTECTED]> wrote: > > Hi, I have worked with nutch for sometime. One thing I am always curious is > when crawling, fetcher's speed will get slower and slower, no matter what > configuration I use. > My last test get this: ( just one site to make the problem more simple) > > OS : winxp > java : 1.6.0.2 > nutch: 0.9 > cpu : AMD 1800 > mem : 1G > network : 3m adsl > > site : wikipedia.org > threads per site :30 > server.delay : 0.5 > > It starts about 6page/s, but reduce to 4 in some minutes, then get slower > and slower. I have run it for 8 hours, just 2page/s left, and it was till > slowing down. > But if I stop it and start one other, it returns full speed (then slows down > again). I am ok with 2 pages/s for one site, but I do hope it will keep that > speed. > > I found there are some guys in this list has the same problem. But I can't > find an answer. > If nutch designed to work this way? > > Thanks! > -- > View this message in context: > http://www.nabble.com/Fetcher-get-slower-and-slower-in-one-run-of-crawling-tf4241580.html#a12069282 > Sent from the Nutch - User mailing list archive at Nabble.com. > >
