Author: Jason de Cordoba
Email: [EMAIL PROTECTED]
Message:
Hi,
While running 10 indexers simultaneously in cache-mode networking within 2 hours to 48
hours will completely stop on the machine.
I cannot access the computer remotely or access the network from the computer and of
course the indexer gets timeouts for the site's its trying to crawl.
I thought perhaps it was all the dns querries so I started up named and had it be its
own dns server but it did not fix it.
This computer just does the crawling for another computer to serve the data, so there
is very little running on it at the moment to interefere with the crawling.
Database is 3.8GB (shows no errors)
amd +1800 w/512MB
freebsd 4.5
mysql 4.0.1
mnoGoSearch-3.1.19
Database statistics
Status Expired Total
-----------------------------
0 1897472 1899198 Not indexed yet
1 0 8 Unknown status
200 0 250278 OK
204 0 143 Unknown status
300 0 89 Multiple Choices
301 0 8120 Moved Permanently
302 0 18993 Moved Temporarily
303 0 3 See Other
304 0 191247 Not Modified
400 0 578 Bad Request
401 0 377 Unauthorized
402 0 1 Payment Required
403 0 752 Forbidden
404 0 37838 Not found
415 0 26 Unsupported Media Type
500 0 661 Internal Server Error
502 0 13 Bad Gateway
503 0 1961 Service Unavailable
504 3 10002 Gateway Timeout
505 0 2 Protocol Version Not Supported
508 0 9 Unknown status
-----------------------------
Total 1897475 2420299
Best Regards,
Jason
ps: Is there anyway to get the indexer to not hit a piticular webserver/ip more than
ten times out of a thousand indexer requests?
Just to be much more considerate of the utilization of other people's webservers.
Reply: <http://www.mnogosearch.org/board/message.php?id=4391>
___________________________________________
If you want to unsubscribe send "unsubscribe general"
to [EMAIL PROTECTED]
