[
https://issues.apache.org/jira/browse/NUTCH-1531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577479#comment-13577479
]
Julien Nioche commented on NUTCH-1531:
--------------------------------------
See also [NUTCH-1314] and [NUTCH-1106]. Your approach would work but ideally we
would like to filter on length regardless of whether the regex-based filter is
used or not. [NUTCH-1314] and [NUTCH-1106] have similar shortcomings. The best
approach would maybe to have that as a new plugin (urlfilter-basic) in the same
way as we have a BasicURLNormalizer and add it to the list of plugins loaded by
default. Can't remember whether the URLFilters are typically called before the
URLNormalizers but assuming they are this would also speed up the normalisation.
Longer term we want to merge the normalization and filtering into a single
logical step.
> URL filtering takes long time for very long URLs
> ------------------------------------------------
>
> Key: NUTCH-1531
> URL: https://issues.apache.org/jira/browse/NUTCH-1531
> Project: Nutch
> Issue Type: Improvement
> Affects Versions: 1.6, 2.1, 1.7, 2.2
> Reporter: Fırat KÜÇÜK
> Priority: Minor
> Fix For: 1.7, 2.2
>
> Attachments: max_url_length.diff, test_case.txt
>
>
> Filtering very long urls (such as base64 image generators) take long time
> (hours). On reducing phase it locks down all the system for hours. Therefore
> some URL length limitation needed. We attached a little patch for this
> improvement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
