[
https://issues.apache.org/jira/browse/NUTCH-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14573108#comment-14573108
]
Luis Lopez commented on NUTCH-2035:
-----------------------------------
Hi [~wastl-nagel], after doing several tests with 100k URLs in my laptop (Core
i7 4650U, 8GB Ram, OpenJDK 1.7.0_79) I can confirm that you are right, the
performance improvement is neglectable, in average I got 19.91s vs 19.54s, for
the url.toLowerCase(). However I got a better improvement shortening the rules
and writing them to be case insensitive (?i). 18.42s.
We are using different regex rules. I'm attaching one of our files.
In any case I think that the rules should be simplified in the default Nutch
regex file. Also, in the tests we can use lower case or case insensitive rules
instead of i.e. "MY.DOMAIN.NAME".
> Regex filter using case sensitive rules.
> ----------------------------------------
>
> Key: NUTCH-2035
> URL: https://issues.apache.org/jira/browse/NUTCH-2035
> Project: Nutch
> Issue Type: Improvement
> Components: plugin
> Affects Versions: 1.10
> Reporter: Luis Lopez
> Priority: Minor
> Labels: filters, regex, regex-urlfilter
> Fix For: 1.11
>
> Attachments: regex-urlfilter.txt
>
>
> Regex expressions are computationally expensive and having “EXE|exe|JPG|jpg”
> etc etc..... adds up if we use complex rules.
> Regex filter should use case insensitive rules to make the rules more
> readable and improve performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
