Author: fasfuuiios
Email: 
Message:
I have noted that even if I start indexer with 5 or 10 or 20 or 40 
threads with CrawlerThreads option in indexer.conf, top command is 
always showing not more than 40% of cpu and very rarely it can rise up 
to 55%. With more threads it can slightly ddos some sites and they 
give 503 error or even 508. Using munin for server monitoring is 
showing rather stable perfomance without high cpu and memory usage 
during indexation. Sometimes indexer hangs but I check it with cron 
each minute and start it again if it is not active.

* * * * *   root        pgrep indexer > /dev/null || 
/usr/local/mnogosearch/sbin/indexer -l

Does mnogosearch has some internal perfomance limitations for indexer 
to make possible parallel searches and indexing? Or maybe I have 
missed something in compiling options or any special options in 
indexer.conf? I have not experimented with more than one indexer 
processes. Is it possible to achieve 80% of cpu usage constantly? If 
yes, what is the safest and stablest way to do it, if server is used 
only for indexing?

Or maybe it is good practice to limit indexer? I have seen php 
crawlers that can easily eat 90% of cpu. Of course, their slow 
perfomance are not compared with mnogosearch high speed. It works very 
fast. But of course, it is interesting how to load server completely 
during indexing.

 

Reply: <http://www.mnogosearch.org/board/message.php?id=21611>

_______________________________________________
General mailing list
General@mnogosearch.org
http://lists.mnogosearch.org/listinfo/general

Reply via email to