John, I know that :) I'm just stating for the reasons of .htaccess and other files its not normal policy to put your ftp path anywhere near your http path or else people could find ways to get at your webserver through ftp..
(especially by stealing cgi programs, looking at your uncompiled php scripts and having the ability to look at those nice config.php and stuff) I don't think its normal practice at all to have your ftp root anywhere near your http root and thus suggest that robots.txt on ftp:// urls is not necessary. ftp security is controlled by the policies of the ftpd daemon as has nothing to do with http protocol. If your referencing a file via http://www.yourdomain.com/yourfile.zip then yeah, use robots.txt, however ftp://yourdomain.com/pub/yourfile.zip won't have a spiders.txt in the root folder in 99.999999% of the cases :) --- [EMAIL PROTECTED] wrote: > On Thu, May 13, 2004 at 09:04:08AM -0700, Byron > Miller wrote: > > FTP by default doesn't use the same file system as > > http and if it did, you wouldn't want to have your > > .htaccess or robots.txt open for all to see what > your > > trying to hide. > > .htaccess and robots.txt are for totally different > purposes. > .htaccess is local to httpd and shouldn't be > accessible by client. > However robots.txt is meant to be accessible by > client and it merely directs > crawlers. robots.txt should not be used to hide > stuff that are not > supposed to be accessible by client. > > John > > > ------------------------------------------------------- > This SF.Net email is sponsored by: SourceForge.net > Broadband > Sign-up now for SourceForge Broadband and get the > fastest > 6.0/768 connection for only $19.95/mo for the first > 3 months! > http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers ------------------------------------------------------- This SF.Net email is sponsored by: SourceForge.net Broadband Sign-up now for SourceForge Broadband and get the fastest 6.0/768 connection for only $19.95/mo for the first 3 months! http://ads.osdn.com/?ad_id=2562&alloc_id=6184&op=click _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
