Thanks Villas. I have read somewhere, even with user-agent set to disallow, search crawlers will still crawl your site. Considering site is live now, I rather just avoid future headache.
On Friday, 5 June 2015 17:39:49 UTC+2, villas wrote: > > Really good +1 ! > > If you are planing to test the app, please do not publish irrelevant or >> uncompleted records...(google remembers everything) :) >> >> > Whilst testing, maybe you should consider a robots.txt > <https://support.google.com/webmasters/answer/6062608?hl=en&ref_topic=6061961&rd=1> > > file with something like this: > > User-agent: * > Disallow: / > -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
