This is utterly fascinating. Does anyone have any idea why Googlebot would try this?
(also - are you sure this isn't something nefarious masquerading as Googlebot?) Jeff On Tue, Jul 5, 2011 at 1:45 PM, Brandon Wirtz <[email protected]> wrote: > User Visits a page that doesn’t have any links from anywhere, and has a > search form on the page.**** > > ** ** > > So the urls that it generates are www.somesite.com/mypage/?s=test**** > > ** ** > > So there is adsense on the page, so the Google Bot finds the page. It says > “OOOoooh Shiny button, let me try” and does so for 3 hours. ALL of that > spike, is Google Bot. 3 hours of asking questions to made up words **** > > ** ** > > Zenate**** > > Zooplus**** > > Zeroxin**** > > Zathura**** > > Xile**** > > Xylose**** > > Yasin **** > > Yasmina**** > > ** ** > > ** ** > > More or less in alphabetical order.**** > > ** ** > > And you can’t throttle it, because if you are on GAE you get assigned a > special crawl rate. By special I mean near infinite. (good most of the > time)**** > > ** ** > > This is technically my fault, didn’t robots.txt to say it couldn’t do > this… and it only cost me like 70 cents… but when it did this the page was > broken, so all pages returned the same nothing. **** > > ** ** > > I have a nice list of things Google thought to try… Not sure what it is > good for, but I have it… **** > > ** ** > > Oh and the broken form it hit the fetch timeout because well the page is > broken… So 8s per hit. Imagine that bill on the new model. **** > > ** ** > > **** > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group at > http://groups.google.com/group/google-appengine?hl=en. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
<<image001.png>>
