[
https://issues.apache.org/jira/browse/NUTCH-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15146271#comment-15146271
]
ASF GitHub Bot commented on NUTCH-2144:
---------------------------------------
Github user thammegowda commented on a diff in the pull request:
https://github.com/apache/nutch/pull/89#discussion_r52835613
--- Diff: src/java/org/apache/nutch/parse/ParseOutputFormat.java ---
@@ -338,6 +340,9 @@ public static String filterNormalize(String fromUrl,
String toUrl,
} catch (MalformedURLException e1) {
return null; // skip it
}
+
+
+
if ("bydomain".equalsIgnoreCase(ignoreExternalLinksMode)) {
String toDomain = URLUtil.getDomainName(targetURL).toLowerCase();
if (toDomain == null || !toDomain.equals(origin)) {
--- End diff --
I did not have a chance to test `db.ignore.external.links.mode : byDomain`
feature.
However, if this new feature should also be covered for overriding, then I
am ready to do so.
> Plugin to override db.ignore.external to exempt interesting external domain
> URLs
> --------------------------------------------------------------------------------
>
> Key: NUTCH-2144
> URL: https://issues.apache.org/jira/browse/NUTCH-2144
> Project: Nutch
> Issue Type: New Feature
> Components: crawldb, fetcher
> Reporter: Thamme Gowda N
> Assignee: Chris A. Mattmann
> Priority: Minor
> Fix For: 1.12
>
> Attachments: ignore-exempt.patch, ignore-exempt.patch
>
>
> Create a rule based urlfilter plugin that allows focused crawler
> (db.ignore.external.links=true) to fetch static resources from external
> domains.
> The generalized version of this: This plugin should permit interesting URLs
> from external domains (by overriding db.ignore.external). The interesting
> urls are decided from a combination of regex and mime-type rules.
> Concrete use case:
> When using Nutch to crawl images from a set of domains, the crawler needs
> to fetch all images which may be linked from CDNs and other domains. In this
> scenario, allowing all external links and then writing hundreds of regular
> expressions is not feasible for large number of domains.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
