[
https://issues.apache.org/jira/browse/LABS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thorsten Scherler updated LABS-198:
-----------------------------------
Summary: NoRobotsClient don't follow the standard for robot.txt (was:
NoRobotsClient don't follow the standar)
> NoRobotsClient don't follow the standard for robot.txt
> ------------------------------------------------------
>
> Key: LABS-198
> URL: https://issues.apache.org/jira/browse/LABS-198
> Project: Labs
> Issue Type: Bug
> Components: Droids
> Reporter: Javier Puerto
> Attachments: norobot-rfc.diff
>
>
> I see that the url for the robots was relative to the path and this not
> follow the robots standard.
> ...
> public static URL findRobotsUrl(URL base, String prefix) throws
> MalformedURLException {
> URL url = new URL(base, "robots.txt");
> boolean exist = existUrl(url);
> ...
> It should be "new URL(base, "/robots.txt");"
> I found this on the web:
> * http://www.robotstxt.org/norobots-rfc.txt (sec 3.1)
> * http://en.wikipedia.org/wiki/Robots.txt
> * http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
> Attach a patch to solve this behavior.
> Salu10.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
