Author: Alexander Barkov
> Hello,
> I've tried this :
> ./indexer -p 0
> but it doesn't work :(
> The indexer sleeps for at least one seconde after each URL.

With -p0 it does not do any delays between URLs.
I guess the bottleneck is in the connection, or in the remote site.

To speed up crawling performance, you can run multiple crawling threads in 
parallel, for example:

indexer -N5

Make sure not to put the the remote site down though.

> It seems impossible to index faster than 1s after each url.
> To index 300 000 document on my website for example, the crawl takes 2 full 
> days !
> Is there a solution ?

Reply: <>

General mailing list

Reply via email to