Lewis John McGibbney commented on NUTCH-1314:

Yes [~wastl-nagel] that was why it was still open. Do you want to port?

> Impose a limit on the length of outlink target urls
> ---------------------------------------------------
>                 Key: NUTCH-1314
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1314
>             Project: Nutch
>          Issue Type: Improvement
>            Reporter: Ferdy Galema
>            Assignee: Lewis John McGibbney
>             Fix For: 2.4
>         Attachments: NUTCH-1314-trunk.patch, NUTCH-1314-v2.patch, 
> NUTCH-1314-v3.patch, NUTCH-1314-v4.patch, NUTCH-1314.patch
> In the past we have encountered situations where crawling specific broken 
> sites resulted in ridiciously long urls that caused the stalling of tasks. 
> The regex plugins (normalizing/filtering) processed single urls for hours, if 
> not indefinitely hanging.
> My suggestion is to limit the outlink url target length as soon possible. It 
> is a configurable limit, the default is 3000. This should be reasonably long 
> enough for most uses. But sufficienly strict enough to make sure regex 
> plugins do not choke on urls that are too long. Please see attached patch for 
> the Nutchgora implementation.
> I'd like to hear what you think about this.

This message was sent by Atlassian JIRA

Reply via email to