On  13:01:15 26/Sep , Jay Pound wrote:
> I crawl with 100 threads, but you need to have 10% available for outbound
> traffic so if your connection is a cablemodem with only 512kb upload you
> will not be able to download at 10mbit but 5mbit, your numbers are low, most

I have also 10Mb in upload..
What machine do you usually use?

> of the machines I run can crawl 30pages a sec with 3mbit 2xT1's and 42pages
> a sec with 9mbit cablemodem connection, just run more threads, that is if
> your hardware can handle it.

but I thought my machine was powerful enought to run 50/100 threads.. I
cannot understand, because the cpu isn't really used, an even memory is
quite unused...

> you can also try and run the -noparse flag and
> parse the data later or on a different faster machine!!!

ok, I'll try with this,but it seems to have no relevant effect..

Thank you!!!

Reply via email to