[
https://issues.apache.org/jira/browse/NUTCH-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146210#comment-15146210
]
ASF GitHub Bot commented on NUTCH-2144:
---------------------------------------
Github user sebastian-nagel commented on a diff in the pull request:
https://github.com/apache/nutch/pull/89#discussion_r52833922
--- Diff: src/plugin/urlfilter-ignoreexempt/README.md ---
@@ -0,0 +1,52 @@
+urlfilter-ignoreexempt
+======================
+ This plugin allows certain urls to be exempted when the external links
are configured to be ignored.
+ This is useful when focused crawl is setup but some resources like
static files are linked from CDNs (external domains).
+
+How to enable ?
+==============
+Add `urlfilter-ignoreexempt` value to `plugin.includes` property
+```xml
+<property>
+ <name>plugin.includes</name>
+ <value>protocol-http|urlfilter-(regex|ignoreexempt)...</value>
+</property>
+```
+
+How to configure rules?
+================
+
+open `conf/db-ignore-external-exemptions.txt` and add rules
+
+#### Format :
+
+```
+UrlRegex1
+UrlRegex2
+UrlRegex3
+```
+
+
+#### NOTE ::
+ 1. If an url matches any of the given regexps then that url is exempted.
+ 2. \# in the beginning makes it a comment line
+ 3. To Test the regex, update this file and use the below command
+ bin/nutch plugin urlfilter-ignoreexempt
org.apache.nutch.urlfilter.ignoreexempt.ExemptionUrlFilter <URL>
+
+
+#### Example :
+
+ To exempt urls ending with image extensions, use this rule
+
+`.*\.(jpg|JPG|png$|PNG|gif|GIF)$# Testing`
--- End diff --
dito
> Plugin to override db.ignore.external to exempt interesting external domain
> URLs
> --------------------------------------------------------------------------------
>
> Key: NUTCH-2144
> URL: https://issues.apache.org/jira/browse/NUTCH-2144
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb, fetcher
> Reporter: Thamme Gowda N
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 1.12
>
> Attachments: ignore-exempt.patch, ignore-exempt.patch
>
>
> Create a rule based urlfilter plugin that allows focused crawler
> (db.ignore.external.links=true) to fetch static resources from external
> domains.
> The generalized version of this: This plugin should permit interesting URLs
> from external domains (by overriding db.ignore.external). The interesting
> urls are decided from a combination of regex and mime-type rules.
> Concrete use case:
> When using Nutch to crawl images from a set of domains, the crawler needs
> to fetch all images which may be linked from CDNs and other domains. In this
> scenario, allowing all external links and then writing hundreds of regular
> expressions is not feasible for large number of domains.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
