Author: rafikCyc Email: Message: I add those Disallow lines so that, both app can crawl the same number of urls (approximatively 140),
Disallow */basket-villeurbanne/author/* Disallow *?p=* Disallow */feed As it seems that Mnogosearch can manage a robots.txt, but not the meta robots noindex,follow.... Here are the results : ----- indexer -C; indexer; [18898]{01} Done (53 seconds, 168 documents, 3503752 bytes, 64.56 Kbytes/sec.) ------ indexer -C; inexer -N5; [19261]{02} Done (15 seconds, 46 documents, 982938 bytes, 63.99 Kbytes/sec.) [19261]{03} Done (15 seconds, 48 documents, 930200 bytes, 60.56 Kbytes/sec.) [19261]{01} Done (5 seconds, 14 documents, 323667 bytes, 63.22 Kbytes/sec.) [19261]{05} Done (15 seconds, 46 documents, 974427 bytes, 63.44 Kbytes/sec.) [19261]{04} Done (5 seconds, 14 documents, 292520 bytes, 57.13 Kbytes/sec.) [19261]{--} Done (26 seconds, 168 documents, 3503752 bytes, 131.60 Kbytes/sec.) indexer -C; indexer -N50; [20289]{11} Done (11 seconds, 28 documents, 585571 bytes, 51.99 Kbytes/sec.) [20289]{28} Done (11 seconds, 29 documents, 705247 bytes, 62.61 Kbytes/sec.) [20289]{16} Done (11 seconds, 30 documents, 635782 bytes, 56.44 Kbytes/sec.) [20289]{30} Done (11 seconds, 30 documents, 635178 bytes, 56.39 Kbytes/sec.) [20289]{--} Done (21 seconds, 168 documents, 3504392 bytes, 162.96 Kbytes/sec.) mysql -uroot -p -N --database=db_test_mnogo --execute="SELECT url FROM url" > ~/ALL.txt; (cat ~/ALL.txt | parallel -j8 --gnu "wget {}"); real 0m10.638s user 0m1.256s sys 0m1.519s --- Screaming Frog : 12s It just confirm the fact that Mnogosearch is relatively slower than Sreaming Frog, and even when i compare to parallel wget bash, mnogosearch is slower. It get little better with indexer -N50 though. Reply: <http://www.mnogosearch.org/board/message.php?id=21767> _______________________________________________ General mailing list General@mnogosearch.org http://lists.mnogosearch.org/listinfo/general