Is the IP of the traffic 66.249.71.88 (as extracted from my logs above)? (I know that when I access my app myself, I get a different IP address which I presume is the one allocated to my wireless router by my ISP.)
Visiting GoogleBot, it looks like they state that they might crawl "once every few seconds". I would expect this frequency when scanning for a first military strike - not for scanning non-offensive commercial web sites. Oh well. Thanks, Brandon. On Nov 17, 8:26 pm, "Brandon Wirtz" <[email protected]> wrote: > Do you have the IP of the traffic? GB Mobile doesn't usually hit things > more than once a day and doesn't typically do a deep crawl, it just checks > if pages are different from the Non-mobile version of the page. But we > recently discovered that the <Insert derogatory swear words> who makes the > Google DOT crawler has been presenting as GB Mobile, in which case the only > way to get it fixed is to stalk him in the parking lot. (I'm only half > kidding. But this guy and the guy who screwed up the Adsense payment > processing should be fired or relocated to Google Siberia and given a dialup > connection to limit how much money they can cost users) > > I am not some where I can easily check but I believe the email for the > Google Dot guy is in the archive, search it, and send him a "QUIT BANGING ON > MY SITE" email. I thought after BlackWater got law enforcement involved the > first time he did this to a bunch of our clients he'd have stopped. > > In case you check the IP and it isn't a Google IP disregard the above > message. Sometimes when an idiot has screwed up enough times I mistakingly > start blaming him for things he didn't actually cause. In those rare cases > the scolding still serves to remind the idiot how big their previous screw > up was and makes me feel better, which helps until the real culprit can be > found. > > -Brandon > > > > > > > > -----Original Message----- > From: [email protected] > > [mailto:[email protected]] On Behalf Of Ian Marshall > Sent: Thursday, November 17, 2011 11:37 AM > To: Google App Engine > Subject: [google-appengine] Googlebot hitting my app once every minute > > Hello Everyone, > > About 26 hours ago, my application (which normally receives almost no > traffic since it is an orphaned developer app) started receiving Googlebot > hits (and a few Microsoft Bing ones) at a rate of around once per minute. > > I can understand trawling daily, but not at this frequency. I am worried > that my other application which has billing enabled might get affected too > in the future. > > Is there anyone I can ask at Google to get this Googlebot frequency reduced > to around once per day? > > I set out a log extract of a few minutes below so that people can see what > is hitting my app. > > 2011-11-17 19:05:21.234 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkAbout::ILinkListener:: 200 257ms 2kb > DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; > +http://www.google.com/bot.html) > > 66.249.71.88 - - [17/Nov/2011:11:05:21 -0800] "GET > /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkAbout::ILinkListener:: HTTP/1.1" 200 > 2539 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; > +http://www.google.com/bot.html)" "[My app ID].appspot.com" > ms=257 cpu_ms=588 api_cpu_ms=238 cpm_usd=0.016715 > instance=00c61b117c68264d536600a9a7ff7f003466ec > > 2011-11-17 19:01:07.517 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:166:lnkHome::ILinkListener:: 200 501ms 2kb > DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; > +http://www.google.com/bot.html) > > 66.249.71.88 - - [17/Nov/2011:11:01:07 -0800] "GET > /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:166:lnkHome::ILinkListener:: HTTP/1.1" 200 > 2543 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; > +http://www.google.com/bot.html)" "[My app ID].appspot.com" > ms=502 cpu_ms=635 api_cpu_ms=238 cpm_usd=0.018012 > instance=00c61b117c68264d536600a9a7ff7f003466ec > > 2011-11-17 18:57:17.929 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:71:lnkLogOn::ILinkListener:: 200 7291ms 2kb > DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; > +http://www.google.com/bot.html) > > 66.249.71.88 - - [17/Nov/2011:10:57:17 -0800] "GET > /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:71:lnkLogOn::ILinkListener:: HTTP/1.1" 200 > 2542 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; > +http://www.google.com/bot.html)" "[My app ID].appspot.com" > ms=7291 cpu_ms=3038 api_cpu_ms=238 cpm_usd=0.084771 loading_request=1 > pending_ms=3544 instance=00c61b117c68264d536600a9a7ff7f003466ec > > 2011-11-17 18:57:15.034 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkHome2::ILinkListener:: 200 662ms 2kb > DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; > +http://www.google.com/bot.html) > > 66.249.71.88 - - [17/Nov/2011:10:57:15 -0800] "GET > /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkHome2::ILinkListener:: HTTP/1.1" 200 > 2541 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; > +http://www.google.com/bot.html)" "[My app ID].appspot.com" > ms=663 cpu_ms=448 api_cpu_ms=238 cpm_usd=0.012826 pending_ms=407 > instance=00c61b117c02585f95ad6983e337a35b3fad33 > > 2011-11-17 18:57:14.204 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkLogOn::ILinkListener:: 200 4556ms 2kb > DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; > +http://www.google.com/bot.html) > > 66.249.71.88 - - [17/Nov/2011:10:57:14 -0800] "GET > /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? > wicket:interface=wicket-0:70:lnkLogOn::ILinkListener:: HTTP/1.1" 200 > 2544 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; > +http://www.google.com/bot.html)" "[My app ID].appspot.com" > ms=4556 cpu_ms=2992 api_cpu_ms=238 cpm_usd=0.083475 loading_request=1 > instance=00c61b117c02585f95ad6983e337a35b3fad33 > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
