Author: Alexander Barkov Email: b...@mnogosearch.org Message: > Thank you for the reply. > > Well, You're right... > With -P0 it does not have a limit of 1s. > > But it remain very slow though. > > -- > > I just did a quick speed test on a small site (500 documents) > Mnogosearch VS screaming frog.
Which site is it? I can try it on my box. > > The results : > > Mnogosearch : 3.2 urls / second > Screaming Frog: 40 urls / second > > Same connection , same remote site, but 10 time faster :( This time difference is very unlikely under the same conditions, unless something is different with configurations. There can be various factors. To mention a few: - Screaming Frog can download in parallel by default. mnoGoSearch downloads with one thread by default. Try "indexer -N10" or "indexer -N30" or "indexer -N50". - Make sure to compare crawlers on empty databases only. Otherwise, crawlers can send "If-Modified-Since" headers. - Make sure crawlers are not using any local caching or local HTTP proxy - Make sure crawlers are not using direct aliases to the local filesystem and perform true HTTP requests. Reply: <http://www.mnogosearch.org/board/message.php?id=21764> _______________________________________________ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general
