[
https://issues.apache.org/jira/browse/NUTCH-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053937#comment-13053937
]
Markus Jelsma edited comment on NUTCH-1011 at 6/23/11 3:55 PM:
---------------------------------------------------------------
The previous regex seems to eat the character preceding the slashes as well.
Here's a new patch using the following expression:
{code}
(?<!:)/{2,}
{code}
was (Author: markus17):
The previous regex seems to eat the character preceding the slashes as
well. Here's a new patch using the following expression:
(?<!:)/{2,}
> Normalize duplicate slashes in URL's
> ------------------------------------
>
> Key: NUTCH-1011
> URL: https://issues.apache.org/jira/browse/NUTCH-1011
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.4, 2.0
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Attachments: NUTCH-1011-all-2.patch, NUTCH-1011-all.patch
>
>
> Many websites produce faulty URL's with multiple slashes e.g.
> http://cocoon.apache.org///////////////////////1.x/dynamic.html
> This can be really nasty if the number of slashes varies, resulting in many
> URL's actually pointing to the same page and generating new (unique) URL's to
> the same or other duplicate pages.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
