David Smith created NUTCH-2973:
----------------------------------
Summary: Single domain names (eg https://localnet) can't be
crawled - filtering fails
Key: NUTCH-2973
URL: https://issues.apache.org/jira/browse/NUTCH-2973
Project: Nutch
Issue Type: Bug
Components: fetcher
Affects Versions: 1.19
Environment: Nutch 1.19, checked on Windows 10 and Ubuntu. Both have
the same issue.
'm trying to crawl a SharePoint intranet using nutch where the URLs are similar
to:
{{https://localnet/something.aspx}}
The issue is that Nutch is rejecting any url with a single element domain name
such as localnet above. "localnet.com" is not rejected, nor is
"local.localnet". It almost feels as if there's a chunk of code within Nutch
that's unrelated to the filtering mechanisms that rejects URLs outright if they
don't have a WWW style format and a WWW-style domain such as .COM
Error message:
{{Total urls rejected by filters: 1}}
I've checked and updated all the _filter_ files in the conf directory. Even
making then incredibly permissive (effectively "crawl everything") has not
helped.
Reporter: David Smith
There appears to be a bug within the core of Nutch that fails to permit any
single domain name URLs to be crawled. Example:
{{https://{*}localnet{*}/something.aspx}}
The issue is that Nutch is rejecting any url with a single element domain name
such as *localnet* above. "localnet.com" is not rejected, nor is
"local.localnet". It almost feels as if there's a chunk of code within Nutch
that's unrelated to the filtering mechanisms that rejects URLs outright if they
don't have a WWW style format and a WWW-style domain such as .COM
Error message:
{{Total urls rejected by filters: 1}}
I've checked and updated all the filter files in the conf directory. Even
making then incredibly permissive (effectively "crawl everything") has not
helped. Immediately that a dot (.) is added to the domain name it is not
rejected - eg blah.localnet.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
