Author: Alexander Barkov
Email: b...@mnogosearch.org
Message:
> Hello,
> 
> I've tried this :
> 
> ./indexer -p 0
> 
> but it doesn't work :(
> The indexer sleeps for at least one seconde after each URL.

With -p0 it does not do any delays between URLs.
I guess the bottleneck is in the connection, or in the remote site.


To speed up crawling performance, you can run multiple crawling threads in 
parallel, for example:

indexer -N5

Make sure not to put the the remote site down though.


> 
> It seems impossible to index faster than 1s after each url.
> 
> To index 300 000 document on my website for example, the crawl takes 2 full 
> days !
> 
> Is there a solution ?

Reply: <http://www.mnogosearch.org/board/message.php?id=21762>

_______________________________________________
General mailing list
General@mnogosearch.org
http://lists.mnogosearch.org/listinfo/general

Reply via email to