robots.txt is a global standard accepted by everyone. Even google, bing use that. I dont think that there is any db file format maintained by web servers for the robots information.
On Tue, Mar 5, 2013 at 1:29 AM, Raja Kulasekaran <[email protected]> wrote: > Hi > > Instead of parsing robots.txt file, why don't ask the web hoster or web > administrator to create the complete parsed text in the db file format at > the robots.txt location itself ? > > Is there are any standard protocol ? It would be a better idea to stop > transferring data through crawlers . > > Please let me know your thoughts on the same . > > Raja >
