I guess it depends on what you are asking to have returned. ( And this bring up another robots.txt question.. below)
http://www.abc.de/xyz Asking for the directory. (where the service is allowed redirection to a temporary default file list or another default file as a reply if the service doesn't wish to send you the whole directory) or http://www.abc.de/xyz/ Asking directly for the file list or default file of that directory. A best practice is to add the slash if you are really asking for the default file list or index for that directory when the default file name is not known. It would be great to know how to ask that http service for the list of "default or index file" names so the agents could verify what file name was indeed associated with the "/" slash. We could then put the file name on the URL to completely qualify that URL path. Anyone? We can scan through all defaults names for each known http services, but almost all I have dealt with have allowed the customization on that default name. The complexity is that the default name can be a list, not a single file name on the service; so the order of checking for the first issued default name is a concern. This is why I would like to know how the agents can query the http services for the default name list, with the returned names listed inorder of operation. Or at least why the web services have not added such a useful service query? ( Was it just not done before, or is there some known security issue) Anyone? - - - - - Crazy thought... This is where the robots.txt file could be used to hold that information for the robot agents that need to know the operational order of the "/" defaults names used on that service. User-agent: * Slash: default.htm, default.html, index.htm, index.html, welcome.html, sitemap.html The above is just for consideration if the robots.txt is ever updated so the robots could be informed of this little detail. -Thomas Kay ---------- Original Text ---------- From: "Matthias Jaekle" <[EMAIL PROTECTED]>, on 21/11/2001 11:49 AM: Hello, I read about adding a slash at the end of the URLs, if there is no absolut path present. But what about pathes ending in subdirectories (xyz). A link to http://www.abc.de/xyz/ might be more correct then the link to http://www.abc.de/xyz But is there a possibility to find out if somebody who was writing http://www.abc.de/xyz is meaning http://www.abc.de/xyz/ In my database of scanned urls I found both versions, so I believe I analysed many files twice. How do I handle this circumstance correctly ? Many thanks for your help Matthias -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]". -- This message was sent by the Internet robots and spiders discussion list ([EMAIL PROTECTED]). For list server commands, send "help" in the body of a message to "[EMAIL PROTECTED]".
