[
https://issues.apache.org/jira/browse/LABS-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638892#action_12638892
]
Javier Puerto commented on LABS-198:
------------------------------------
>> It should be "new URL(base, "/robots.txt");"
>In your patch however you do "new URL(getUrlPrefix(base) + "/robots.txt");".
This is the first version.
>The first suggestion is correct the patch is not.
I change the code while wrote the comment and forget to change the attachment.
Salu2.
> NoRobotsClient does not follow the standard for robot.txt location
> ------------------------------------------------------------------
>
> Key: LABS-198
> URL: https://issues.apache.org/jira/browse/LABS-198
> Project: Labs
> Issue Type: Bug
> Components: Droids
> Reporter: Javier Puerto
> Attachments: norobot-rfc.diff
>
>
> I see that the url for the robots was relative to the path and this not
> follow the robots standard.
> ...
> public static URL findRobotsUrl(URL base, String prefix) throws
> MalformedURLException {
> URL url = new URL(base, "robots.txt");
> boolean exist = existUrl(url);
> ...
> It should be "new URL(base, "/robots.txt");"
> I found this on the web:
> * http://www.robotstxt.org/norobots-rfc.txt (sec 3.1)
> * http://en.wikipedia.org/wiki/Robots.txt
> * http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
> Attach a patch to solve this behavior.
> Salu10.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
