Do you have the IP of the traffic? GB Mobile doesn't usually hit things more than once a day and doesn't typically do a deep crawl, it just checks if pages are different from the Non-mobile version of the page. But we recently discovered that the <Insert derogatory swear words> who makes the Google DOT crawler has been presenting as GB Mobile, in which case the only way to get it fixed is to stalk him in the parking lot. (I'm only half kidding. But this guy and the guy who screwed up the Adsense payment processing should be fired or relocated to Google Siberia and given a dialup connection to limit how much money they can cost users)
I am not some where I can easily check but I believe the email for the Google Dot guy is in the archive, search it, and send him a "QUIT BANGING ON MY SITE" email. I thought after BlackWater got law enforcement involved the first time he did this to a bunch of our clients he'd have stopped. In case you check the IP and it isn't a Google IP disregard the above message. Sometimes when an idiot has screwed up enough times I mistakingly start blaming him for things he didn't actually cause. In those rare cases the scolding still serves to remind the idiot how big their previous screw up was and makes me feel better, which helps until the real culprit can be found. -Brandon -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Ian Marshall Sent: Thursday, November 17, 2011 11:37 AM To: Google App Engine Subject: [google-appengine] Googlebot hitting my app once every minute Hello Everyone, About 26 hours ago, my application (which normally receives almost no traffic since it is an orphaned developer app) started receiving Googlebot hits (and a few Microsoft Bing ones) at a rate of around once per minute. I can understand trawling daily, but not at this frequency. I am worried that my other application which has billing enabled might get affected too in the future. Is there anyone I can ask at Google to get this Googlebot frequency reduced to around once per day? I set out a log extract of a few minutes below so that people can see what is hitting my app. 2011-11-17 19:05:21.234 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkAbout::ILinkListener:: 200 257ms 2kb DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 66.249.71.88 - - [17/Nov/2011:11:05:21 -0800] "GET /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkAbout::ILinkListener:: HTTP/1.1" 200 2539 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; +http://www.google.com/bot.html)" "[My app ID].appspot.com" ms=257 cpu_ms=588 api_cpu_ms=238 cpm_usd=0.016715 instance=00c61b117c68264d536600a9a7ff7f003466ec 2011-11-17 19:01:07.517 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:166:lnkHome::ILinkListener:: 200 501ms 2kb DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 66.249.71.88 - - [17/Nov/2011:11:01:07 -0800] "GET /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:166:lnkHome::ILinkListener:: HTTP/1.1" 200 2543 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; +http://www.google.com/bot.html)" "[My app ID].appspot.com" ms=502 cpu_ms=635 api_cpu_ms=238 cpm_usd=0.018012 instance=00c61b117c68264d536600a9a7ff7f003466ec 2011-11-17 18:57:17.929 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:71:lnkLogOn::ILinkListener:: 200 7291ms 2kb DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 66.249.71.88 - - [17/Nov/2011:10:57:17 -0800] "GET /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:71:lnkLogOn::ILinkListener:: HTTP/1.1" 200 2542 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; +http://www.google.com/bot.html)" "[My app ID].appspot.com" ms=7291 cpu_ms=3038 api_cpu_ms=238 cpm_usd=0.084771 loading_request=1 pending_ms=3544 instance=00c61b117c68264d536600a9a7ff7f003466ec 2011-11-17 18:57:15.034 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkHome2::ILinkListener:: 200 662ms 2kb DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 66.249.71.88 - - [17/Nov/2011:10:57:15 -0800] "GET /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkHome2::ILinkListener:: HTTP/1.1" 200 2541 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; +http://www.google.com/bot.html)" "[My app ID].appspot.com" ms=663 cpu_ms=448 api_cpu_ms=238 cpm_usd=0.012826 pending_ms=407 instance=00c61b117c02585f95ad6983e337a35b3fad33 2011-11-17 18:57:14.204 /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkLogOn::ILinkListener:: 200 4556ms 2kb DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 66.249.71.88 - - [17/Nov/2011:10:57:14 -0800] "GET /;jsessionid=OgMqQSTfpLQ4XYJTwHb09Q? wicket:interface=wicket-0:70:lnkLogOn::ILinkListener:: HTTP/1.1" 200 2544 - "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/ 2.1; +http://www.google.com/bot.html)" "[My app ID].appspot.com" ms=4556 cpu_ms=2992 api_cpu_ms=238 cpm_usd=0.083475 loading_request=1 instance=00c61b117c02585f95ad6983e337a35b3fad33 -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en. -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
