[
https://issues.apache.org/jira/browse/NUTCH-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472015#comment-16472015
]
Sebastian Nagel commented on NUTCH-2468:
----------------------------------------
Hi [[email protected]], I assume you are referring to my comment on the [user
mailing
list|https://lists.apache.org/thread.html/24de5f0e9b7bc1013f6a0205d1c8d416935ba2c4999c9b0acaba2116@%3Cuser.nutch.apache.org%3E]:
bq. By default filters are only applied to:
- injected URLs
- outlinks during parsing
- redirects (if fetcher follows redirects)
That's how the script bin/crawl behaves in combination with default settings.
Yes, please open a separate issue to change the behavior of Generator. Since
very old days it was written to filter by default, and NUTCH-403 added the
option not to filter. Generator should instead provide the option -filter but
do not filter by default. In consequence, there is no need to set it explicitly
in bin/crawl. Thanks!
> should filter out invalid URLs by default
> -----------------------------------------
>
> Key: NUTCH-2468
> URL: https://issues.apache.org/jira/browse/NUTCH-2468
> Project: Nutch
> Issue Type: Improvement
> Components: bin
> Affects Versions: 1.12
> Reporter: Michael Coffey
> Priority: Minor
> Fix For: 2.4, 1.14
>
>
> Some Nutch components, by default, should reject invalid URLs. This was
> recently discussed in the users mailing list and has affected my work for a
> while. Although there may be some special-purpose needs to collect invalid
> URLs, they are not generally useful for crawling.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
