Barry Warsaw wrote:
On Aug 31, 2009, at 1:15 PM, C Nulk wrote:
As for using robots.txt, hmm, it is not the legitimate search
engines I
care about, it is the search engines/crawlers that do not respect my
robots.txt file that I care about. If I had an effective way to
consistently
Mark Sapiro wrote:
Barry Warsaw wrote:
On Aug 31, 2009, at 1:15 PM, C Nulk wrote:
As for using robots.txt, hmm, it is not the legitimate search
engines I
care about, it is the search engines/crawlers that do not respect my
robots.txt file that I care about. If I had an
