Author: Alexander Barkov
> Thank you for the reply.
> Well, You're right...
> With -P0 it does not have a limit of 1s.
> But it remain very slow though.
> --
> I just did a quick speed test on a small site (500 documents)
> Mnogosearch VS screaming frog.

Which site is it? I can try it on my box.

> The results :
> Mnogosearch : 3.2 urls / second
> Screaming Frog: 40 urls / second
> Same connection , same remote site, but 10 time faster :(

This time difference is very unlikely under the same conditions,
unless something is different with configurations.

There can be various factors. To mention a few:

- Screaming Frog can download in parallel by default.
  mnoGoSearch downloads with one thread by default.
  Try "indexer -N10" or "indexer -N30" or "indexer -N50".
- Make sure to compare crawlers on empty databases only.
  Otherwise, crawlers can send "If-Modified-Since" headers.
- Make sure crawlers are not using any local caching or local HTTP proxy
- Make sure crawlers are not using direct aliases to the local filesystem and 
perform true HTTP requests.

Reply: <>

General mailing list

Reply via email to