Not all bots will necessarily pick up robots.txt changes within a couple days. But if these issues persist, then maybe they're indicating that your site is popular—probably a good thing, even if you now have some work to do. Unless you're in the mood to spend more money, set up the caching that I mentioned before; it's the most effective way to optimize MediaWiki's performance without adding more hardware.
On Mon, Feb 2, 2009 at 10:20 PM, Philip Beach <[email protected]>wrote: > Okay, I have ruled out bots. After disallowing all user-agents for a few > days the load did not decrease, my site continued having errors. I have > also, with the help of my host, ruled out DOS attacks. I have also > optimized > the database. I don't know what else to try, my users are still getting > this > error every couple of minutes. > > Service Temporarily Unavailable > The server is temporarily unable to service your request due to maintenance > downtime or capacity problems. Please try again later. > > Additionally, a 404 Not Found error was encountered while trying to use an > ErrorDocument to handle the request. > > It seems when analyzing the load factor compared to the time the errors > appear that the load steadily increases to a factor of about 25, then the > error appears, the load decreases, and it goes back online. My host said > this was the way that the server behaves when under a heavy load of legit > users and that I should focus on optimizing the database. However, this is > a > brand new quad-core server. My old server (dual core, half the ram) was > running the same exact database with no problems. > > Does anyone have any ideas? > > On Mon, Feb 2, 2009 at 3:34 AM, Benjamin Lees <[email protected]> > wrote: > > > Google offers such a tool with their Webmaster Tools < > > https://www.google.com/webmasters/tools>; I'm sure there are plenty of > > similar tools out there. > > > > On Sun, Feb 1, 2009 at 1:17 PM, Philip Beach <[email protected]> > > wrote: > > > > > Ok thanks, but how can I be sure it's working before all my pages drop > > > off google (if it were wrong). Is there some way to validate it in > > > the context of my site? > > > > > > Thanks again > > > > > > > > > On 2/1/09, Benjamin Lees <[email protected]> wrote: > > > > TryUser-agent: * > > > > Disallow: /index.php > > > > Disallow: /skins/ > > > > Disallow: /Special:Search > > > > Disallow: /Special:Random > > > > > > > > Some other good rules to include are > > > > Disallow: /MediaWiki: > > > > Disallow: /Template: > > > > > > > > and maybe > > > > Disallow: /Category: > > > > > > > > On Sun, Feb 1, 2009 at 2:06 AM, Philip Beach <[email protected] > > > > > wrote: > > > > > > > >> Haha true about rotting RAM, I'll look into that. I am not using > > > wikimedia > > > >> style URL's, sadly :( it just didn't happen when the site was first > > set > > > up > > > >> and I can't move it now, for various reasons. All of my files are in > > the > > > >> web-root /. However, through an apache alias, my url is > > > >> mywiki.com/Pagename. > > > >> > > > >> How would robots.txt look for that? Would I simply drop the > preceeding > > > >> /wiki, like this? > > > >> > > > >> User-agent: * > > > >> Disallow: /Special:Search > > > >> Disallow: /Special:Random > > > >> > > > >> Thanks a ton! > > > >> > > > >> On Sun, Feb 1, 2009 at 1:47 AM, Benjamin Lees <[email protected] > > > > > >> wrote: > > > >> > > > >> > Assuming you have Wikimedia-style URLs: > > > >> > User-agent: * > > > >> > Disallow: /w/ > > > >> > Disallow: /wiki/Special:Search > > > >> > Disallow: /wiki/Special:Random > > > >> > > > > >> > Your server will be able to handle a lot more if you set up as > much > > > >> caching > > > >> > as you can <http://www.mediawiki.org/wiki/Manual:Cache>. No > sense > > > >> letting > > > >> > all that spare RAM rot. :) > > > >> > > > > >> > > > > >> > On Sat, Jan 31, 2009 at 10:02 PM, Philip Beach < > > > [email protected] > > > >> > >wrote: > > > >> > > > > >> > > I already have checked the access logs. It appears that Google > and > > > >> Yahoo > > > >> > > are > > > >> > > indeed generating a lot of traffic. Good idea Rob, I've been > > working > > > >> > > on > > > >> > > this > > > >> > > for a while. > > > >> > > > > > >> > > Just out of curiosity, what should my robots.txt look like for > > > >> Mediawiki. > > > >> > > Does anything need to be disallowed? > > > >> > > > > > >> > > On Sat, Jan 31, 2009 at 8:30 PM, Platonides < > [email protected] > > > > > > >> > wrote: > > > >> > > > > > >> > > > You should check the access logs for which is causing the > error. > > > >> > > > > > > >> > > > > > > >> > > > _______________________________________________ > > > >> > > > MediaWiki-l mailing list > > > >> > > > [email protected] > > > >> > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > >> > > > > > > >> > > _______________________________________________ > > > >> > > MediaWiki-l mailing list > > > >> > > [email protected] > > > >> > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > >> > > > > > >> > _______________________________________________ > > > >> > MediaWiki-l mailing list > > > >> > [email protected] > > > >> > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > >> > > > > >> _______________________________________________ > > > >> MediaWiki-l mailing list > > > >> [email protected] > > > >> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > >> > > > > _______________________________________________ > > > > MediaWiki-l mailing list > > > > [email protected] > > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > > > > > > _______________________________________________ > > > MediaWiki-l mailing list > > > [email protected] > > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > > > _______________________________________________ > > MediaWiki-l mailing list > > [email protected] > > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > > > _______________________________________________ > MediaWiki-l mailing list > [email protected] > https://lists.wikimedia.org/mailman/listinfo/mediawiki-l > _______________________________________________ MediaWiki-l mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
