On Thu, May 13, 2004 at 09:04:08AM -0700, Byron Miller wrote: > FTP by default doesn't use the same file system as > http and if it did, you wouldn't want to have your > .htaccess or robots.txt open for all to see what your > trying to hide.
.htaccess and robots.txt are for totally different purposes. .htaccess is local to httpd and shouldn't be accessible by client. However robots.txt is meant to be accessible by client and it merely directs crawlers. robots.txt should not be used to hide stuff that are not supposed to be accessible by client. John ------------------------------------------------------- This SF.Net email is sponsored by: SourceForge.net Broadband Sign-up now for SourceForge Broadband and get the fastest 6.0/768 connection for only $19.95/mo for the first 3 months! http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
