On 13:01:15 26/Sep , Jay Pound wrote: > I crawl with 100 threads, but you need to have 10% available for outbound > traffic so if your connection is a cablemodem with only 512kb upload you > will not be able to download at 10mbit but 5mbit, your numbers are low, most
I have also 10Mb in upload.. What machine do you usually use? > of the machines I run can crawl 30pages a sec with 3mbit 2xT1's and 42pages > a sec with 9mbit cablemodem connection, just run more threads, that is if > your hardware can handle it. but I thought my machine was powerful enought to run 50/100 threads.. I cannot understand, because the cpu isn't really used, an even memory is quite unused... > you can also try and run the -noparse flag and > parse the data later or on a different faster machine!!! ok, I'll try with this,but it seems to have no relevant effect.. Thank you!!!
