[
https://issues.apache.org/jira/browse/NUTCH-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15142021#comment-15142021
]
ASF GitHub Bot commented on NUTCH-2144:
---------------------------------------
GitHub user thammegowda opened a pull request:
https://github.com/apache/nutch/pull/89
NUTCH-2144 : override db.ignore.external to exempt interesting external
domain URLs
+ Add extension point
org.apache.nutch.net.URLExemptionFilter
+ Modify FetcherThread and ParseOutputFormat to
integrate new extension point
+ Add extension urlfilter-ignoreexempt
+ build configs modified to include new extension
Resolves https://issues.apache.org/jira/browse/NUTCH-2144
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/thammegowda/nutch NUTCH-2144-ignore-exempt
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nutch/pull/89.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #89
----
commit 29c7ae4ec088f0f428ab95992e10af9d87a231ad
Author: Thamme Gowda <[email protected]>
Date: 2015-10-19T14:31:04Z
Add an extension point and an extension to
override 'db.ignore.external.links'.
+ Add extension point
org.apache.nutch.net.URLExemptionFilter
+ Modify FetcherThread and ParseOutputFormat to
integrate new extension point
+ Add extension urlfilter-ignoreexempt
+ build configs modified to include new extension
commit 43583f7af19c21dd7553c5e41c026cb852f0cfa1
Author: Thamme Gowda <[email protected]>
Date: 2016-02-11T00:03:03Z
Added ignore exemption extension point and an extension
commit 559eb4d905a081e34095f0e9d5ee3e805363ccc3
Author: Thamme Gowda <[email protected]>
Date: 2016-02-11T00:21:08Z
README updated
commit 3cf887befc56c3cf4127f45385e83c47248dd6e9
Author: Thamme Gowda <[email protected]>
Date: 2016-02-11T00:23:12Z
Add an example rule fiile
----
> Plugin to override db.ignore.external to exempt interesting external domain
> URLs
> --------------------------------------------------------------------------------
>
> Key: NUTCH-2144
> URL: https://issues.apache.org/jira/browse/NUTCH-2144
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb, fetcher
> Reporter: Thamme Gowda N
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 1.12
>
> Attachments: ignore-exempt.patch, ignore-exempt.patch
>
>
> Create a rule based urlfilter plugin that allows focused crawler
> (db.ignore.external.links=true) to fetch static resources from external
> domains.
> The generalized version of this: This plugin should permit interesting URLs
> from external domains (by overriding db.ignore.external). The interesting
> urls are decided from a combination of regex and mime-type rules.
> Concrete use case:
> When using Nutch to crawl images from a set of domains, the crawler needs
> to fetch all images which may be linked from CDNs and other domains. In this
> scenario, allowing all external links and then writing hundreds of regular
> expressions is not feasible for large number of domains.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
