I think GoogleBot Punches through edge cache too. I can see the peaks that are caused by the Google Bot hitting my systems, but in the request types my dynamic requests go up, but my cached requests stay where they were. Even if Google Bot was not getting quite the ratios users were, you'd expect there to be a lift during these times. I mean the Bot raises traffic 10 fold for an hour, you'd think I'd see 20% lift in cached requests during this time.
I'm mostly just "venting" since I know the Crawl team won't see this, and my guy over there has long since moved on. but just something I noticed. From: [email protected] [mailto:[email protected]] On Behalf Of Plumo Sent: Sunday, November 20, 2011 6:23 AM To: [email protected] Subject: [google-appengine] Re: Google Bot Is Your enemy Part 2 pity the Google Bot doesn't support Crawl-delay in robots.txt -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/68trxyxxuaMJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
