Hi, I meant to move the entire crawl process in the client environment , create "robots.db" and fetch only robots.db as a indexed data .
Raja On Tue, Mar 5, 2013 at 8:27 PM, Tejas Patil <[email protected]>wrote: > robots.txt is a global standard accepted by everyone. Even google, bing use > that. I dont think that there is any db file format maintained by web > servers for the robots information. > > > On Tue, Mar 5, 2013 at 1:29 AM, Raja Kulasekaran <[email protected]> > wrote: > > > Hi > > > > Instead of parsing robots.txt file, why don't ask the web hoster or web > > administrator to create the complete parsed text in the db file format at > > the robots.txt location itself ? > > > > Is there are any standard protocol ? It would be a better idea to stop > > transferring data through crawlers . > > > > Please let me know your thoughts on the same . > > > > Raja > > >
