[
https://issues.apache.org/jira/browse/NUTCH-2689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16748765#comment-16748765
]
Sebastian Nagel commented on NUTCH-2689:
----------------------------------------
The benchmark times before ...
{noformat}
% grep 'bench time' build/urlfilter-regex/test/*.txt
2019-01-22 14:23:42,772 INFO api.RegexURLFilterBaseTest ... - bench time (50)
146ms
2019-01-22 14:23:42,946 INFO api.RegexURLFilterBaseTest ... - bench time (100)
171ms
2019-01-22 14:23:43,195 INFO api.RegexURLFilterBaseTest ... - bench time (200)
248ms
2019-01-22 14:23:43,680 INFO api.RegexURLFilterBaseTest ... - bench time (400)
485ms
2019-01-22 14:23:44,574 INFO api.RegexURLFilterBaseTest ... - bench time (800)
893ms
% grep 'bench time' build/urlfilter-automaton/test/*.txt
2019-01-22 14:24:17,793 INFO api.RegexURLFilterBaseTest ... - bench time (50)
136ms
2019-01-22 14:24:17,894 INFO api.RegexURLFilterBaseTest ... - bench time (100)
97ms
2019-01-22 14:24:18,071 INFO api.RegexURLFilterBaseTest ... - bench time (200)
177ms
2019-01-22 14:24:18,324 INFO api.RegexURLFilterBaseTest ... - bench time (400)
252ms
2019-01-22 14:24:18,761 INFO api.RegexURLFilterBaseTest ... - bench time (800)
436ms
{noformat}
... and after the improvements (see PR):
{noformat}
% grep 'bench time' build/urlfilter-regex/test/*.txt
2019-01-22 15:14:19,886 INFO api.RegexURLFilterBaseTest ... - bench time (50)
232ms
2019-01-22 15:14:20,017 INFO api.RegexURLFilterBaseTest ... - bench time (100)
126ms
2019-01-22 15:14:20,158 INFO api.RegexURLFilterBaseTest ... - bench time (200)
140ms
2019-01-22 15:14:20,385 INFO api.RegexURLFilterBaseTest ... - bench time (400)
227ms
2019-01-22 15:14:20,794 INFO api.RegexURLFilterBaseTest ... - bench time (800)
409ms
% grep 'bench time' build/urlfilter-automaton/test/*.txt
2019-01-22 15:14:37,708 INFO api.RegexURLFilterBaseTest ... - bench time (50)
48ms
2019-01-22 15:14:37,752 INFO api.RegexURLFilterBaseTest ... - bench time (100)
41ms
2019-01-22 15:14:37,821 INFO api.RegexURLFilterBaseTest ... - bench time (200)
69ms
2019-01-22 15:14:37,914 INFO api.RegexURLFilterBaseTest ... - bench time (400)
93ms
2019-01-22 15:14:38,080 INFO api.RegexURLFilterBaseTest ... - bench time (800)
165ms
{noformat}
There is some variation when the benchmarks are repeated, but the performance
increase seems significant.
> Speed up urlfilter-regex and urlfilter-automaton
> ------------------------------------------------
>
> Key: NUTCH-2689
> URL: https://issues.apache.org/jira/browse/NUTCH-2689
> Project: Nutch
> Issue Type: Improvement
> Components: plugin
> Affects Versions: 1.15
> Reporter: Sebastian Nagel
> Assignee: Sebastian Nagel
> Priority: Minor
> Fix For: 1.16
>
>
> The unit tests of urlfilter-regex and urlfilter-automaton include a
> benchmark. After playing and benchmarking modifications the following changes
> seem to significantly improve the performance:
> - do not extract host and domain name from the URL if not needed (no
> host/domain-specific rules used, cf. NUTCH-1838)
> - use non-capturing groups if possible
> - use {{(?i)}} to make the patterns case insensitive and remove uppercase
> variants
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
