On Sat, Jul 27, 2013 at 10:41 AM, timh <[email protected]> wrote: > Looking at the user agent (assuming I am interpreting it correctly) you > are being hit by google bot. Its probably hitting you very fast and each > request can not be serviced so it causes heaps of new instances to be > started to service the requests. > > I have had similiar problems in the past from google bot (though not > mobile) and was able to get my site flagged so that google bot wouldn't hit > it so hard and fast. > > That might be worth looking at - where the/what user request is prompting > the instance start requests. I find the only times where I get instances > started rapidly are when a bot hits me. > The main culprit at the moment is Sosospider/2.0; +http://help.soso.com/ > webspider.htm) >
Google Bot supports the crawl-delay parameter in robots.txt ( http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive ); essentially you can tell it to wait x amount of seconds per page request. Most robots support the parameter, it might be worth it for OP to declare a crawl delay and see if it reduces the amount of instances spinning up (for other search engine robots as well). ----------------- -Vinny P Technology & Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
