So, while Google bots are sticking their noses into everyone's online business, they are unable to tell if/when they come across their own widgeted boxes? Cool ;))))
On Jul 5, 9:35 pm, "Brandon Wirtz" <[email protected]> wrote: > Nah, usually it does it on a smaller scale. The idea is to find if you have > anything interesting in your query box. > > http://googlewebmastercentral.blogspot.com/2008/04/crawling-through-h... > ms.html > > From: [email protected] > [mailto:[email protected]] On Behalf Of Jeff Schnitzer > Sent: Tuesday, July 05, 2011 5:32 PM > To: [email protected] > Subject: Re: [google-appengine] Got to love Google Bot > > This is utterly fascinating. Does anyone have any idea why Googlebot would > try this? > > (also - are you sure this isn't something nefarious masquerading as > Googlebot?) > > Jeff > > On Tue, Jul 5, 2011 at 1:45 PM, Brandon Wirtz <[email protected]> wrote: > > User Visits a page that doesn't have any links from anywhere, and has a > search form on the page. > > So the urls that it generates arewww.somesite.com/mypage/?s=test > > So there is adsense on the page, so the Google Bot finds the page. It says > "OOOoooh Shiny button, let me try" and does so for 3 hours. ALL of that > spike, is Google Bot. 3 hours of asking questions to made up words > > Zenate > > Zooplus > > Zeroxin > > Zathura > > Xile > > Xylose > > Yasin > > Yasmina > > More or less in alphabetical order. > > And you can't throttle it, because if you are on GAE you get assigned a > special crawl rate. By special I mean near infinite. (good most of the > time) > > This is technically my fault, didn't robots.txt to say it couldn't do this. > and it only cost me like 70 cents. but when it did this the page was > broken, so all pages returned the same nothing. > > I have a nice list of things Google thought to try. Not sure what it is good > for, but I have it. > > Oh and the broken form it hit the fetch timeout because well the page is > broken. So 8s per hit. Imagine that bill on the new model. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected] > <mailto:google-appengine%[email protected]> . > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/google-appengine?hl=en. > > image001.png > 43KViewDownload -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
