[
https://issues.apache.org/jira/browse/NUTCH-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17621023#comment-17621023
]
Markus Jelsma commented on NUTCH-2973:
--------------------------------------
Hello David,
By default urlfilter-validator is an active plugin, and it will reject
everything localhost, or localnet. You can disable the plugin in your
plugin.includes configuration directive.
> Single domain names (eg https://localnet) can't be crawled - filtering fails
> ----------------------------------------------------------------------------
>
> Key: NUTCH-2973
> URL: https://issues.apache.org/jira/browse/NUTCH-2973
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.19
> Environment: Nutch 1.19, checked on Windows 10 and Ubuntu. Both have
> the same issue.
> 'm trying to crawl a SharePoint intranet using nutch where the URLs are
> similar to:
>
> {{https://localnet/something.aspx}}
> The issue is that Nutch is rejecting any url with a single element domain
> name such as localnet above. "localnet.com" is not rejected, nor is
> "local.localnet". It almost feels as if there's a chunk of code within Nutch
> that's unrelated to the filtering mechanisms that rejects URLs outright if
> they don't have a WWW style format and a WWW-style domain such as .COM
> Error message:
>
> {{Total urls rejected by filters: 1}}
> I've checked and updated all the _filter_ files in the conf directory. Even
> making then incredibly permissive (effectively "crawl everything") has not
> helped.
> Reporter: David Smith
> Priority: Blocker
>
> There appears to be a bug within the core of Nutch that fails to permit any
> single domain name URLs to be crawled. Example:
> {{https://{*}localnet{*}/something.aspx}}
> The issue is that Nutch is rejecting any url with a single element domain
> name such as *localnet* above. "localnet.com" is not rejected, nor is
> "local.localnet". It almost feels as if there's a chunk of code within Nutch
> that's unrelated to the filtering mechanisms that rejects URLs outright if
> they don't have a WWW style format and a WWW-style domain such as .COM
> Error message:
> {{Total urls rejected by filters: 1}}
> I've checked and updated all the filter files in the conf directory. Even
> making then incredibly permissive (effectively "crawl everything") has not
> helped. Immediately that a dot (.) is added to the domain name it is not
> rejected - eg blah.localnet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
