[
https://issues.apache.org/jira/browse/NUTCH-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571680#comment-14571680
]
Sebastian Nagel commented on NUTCH-2035:
----------------------------------------
Thanks, [~betolink]!
* is it possible to substantiate "computationally expensive"? What CPU time is
saved by the patch for which Java version, URL sample, etc.
* I've run a quick test using 100k URLs and
{noformat}
time $NUTCH_HOME/bin/nutch plugin urlfilter-regex
org.apache.nutch.urlfilter.regex.RegexURLFilter <100k_urls.txt >/dev/null
{noformat}
with 3 runs per config/test and taking the shortest time for each configuration
got only minimal savings for the patched trial: 11.94 sec. vs 11.97 (user
time). That's not really significant (agreed, my trivial measurement isn't
significant, too).
* currently the user is able to decide whether to match case sensitive or not.
Shall we really remove this and make the regex url filter be always case
insensitive?
* if it's about readability of the patterns, why not change the pattern to:
{noformat}
-(?i)\.(gif|jpg|png|ico|css|sit|eps|wmf|zip|ppt|mpg|xls|gz|rpm|tgz|mov|exe|jpeg|bmp|js)$
{noformat}
Btw., this variant seems to win the race: 11.52 sec.
* in general, the plugin urlfilter-suffix is much faster to filter by suffix
patterns, esp. if the list gets longer. There is already a hint in to
conf/regex-urlfilter.txt.template
regarding this point.
> Regex filter using case sensitive rules.
> ----------------------------------------
>
> Key: NUTCH-2035
> URL: https://issues.apache.org/jira/browse/NUTCH-2035
> Project: Nutch
> Issue Type: Improvement
> Components: plugin
> Affects Versions: 1.10
> Reporter: Luis Lopez
> Priority: Minor
> Labels: filters, regex, regex-urlfilter
> Fix For: 1.11
>
>
> Regex expressions are computationally expensive and having “EXE|exe|JPG|jpg”
> etc etc..... adds up if we use complex rules.
> Regex filter should use case insensitive rules to make the rules more
> readable and improve performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
