Well yes, as a webmaster you do have to decide if you willing to 'pay' for crewlers wanting to access your site.
Its a tradeoff, the 'cost' verses the potential benefit of having your site included in their search engine. You could opt to proactivly manage crawler activity, in the first case with robots.txt, but also by shaping the responces you give (eg robots noindex/nofollow or even <a rel=nofollow etc) to help persade bots to only crawl the pages you interested in having potentially indexed. Or even just redeisinging your link structure to make it more economic for bots to crawl. Or if you think the 'bots' are potentially malicious, or otherwise interferring with the site, can take even more active measures and block them. Services like cloudflare will help block them. Finally just making sure your site is optimised, eg http://www.webpagetest.org can help. In terms of Appengine, making sure static assets are served as such (so dont use up instance power) even bots will retrieve some static assets. ... there's no magic 'fix', have to decide on your own priorities. On 7 November 2013 16:19, Martin Descours <[email protected]> wrote: > > Hello, > > yesterday my app was visited 35 times by HUMANS. It seems however that a > machine was crawling the website. > I was overquota in a few hours. Today i pay max 5USD per day. for 35 real > people it seems way too much. > > I dont feel really good paying for crawlers that block the access to my > website to regular users > > What can i do to invest money in the real users instead of crawlers ? (and > i am not talking about not referencing my app) > > app : www.conceptstore.me > > > Thank you for your help ! > > > M. > > -- > You received this message because you are subscribed to the Google Groups > "Google App Engine" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at http://groups.google.com/group/google-appengine. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/groups/opt_out.
