[
https://issues.apache.org/jira/browse/NUTCH-546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12522941
]
Marc Brette commented on NUTCH-546:
-----------------------------------
Sounds good. The URL syntax is defined in RFC 1738
http://www.ietf.org/rfc/rfc1738.txt
I don't know the design decision behind UrlValidator, but why didn't you just
instanciate the java URL class ?
As for the 'space' character issue, it is the responsibility of the
protocol-file plugin to ensure correct encoding, but shouldn't we be more
flexible ?
Isn't it possible that we encounter such issue with URL in pages? (you may also
claim that it would be more friendly to allow the administrator not to have to
encode this character in the configuration file).
I didn't start working on a solution Doğacan, thanks for looking into this.
> file URL are filtered out by the crawler
> ----------------------------------------
>
> Key: NUTCH-546
> URL: https://issues.apache.org/jira/browse/NUTCH-546
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.0.0
> Environment: Windows XP
> Nutch trunk from Monday, August 20th 2007
> Reporter: Marc Brette
>
> I tried to index file system using the file:/ protocol, which worked fine in
> version 0.9
> The file URL are being filtered out and not fetched at all.
> I investigated the code and saw that there are 2 issues:
> 1) One is with the class UrlValidator: when validating an URL, it check the
> 'authority', a combination of host and port. As it is null for file, the URL
> is rejected.
> 2) Once this check is removed, files that contain space characters (and maybe
> other characters to be URL encoded) are also filtered out. It maybe be
> because the file protocol plugin doesn't URL encode space characters and/or
> UrlValidator is enforce the rule to encode such character.
> To workaround these issues, I just commented out UrlValidator checks and it
> works fine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
