Gerard Bouchar created NUTCH-2554:

             Summary: parsechecker can't fetch some URLs
                 Key: NUTCH-2554
             Project: Nutch
          Issue Type: Bug
            Reporter: Gerard Bouchar

The parserchecker (org.apache.nutch.parse.ParserChecker) calls 
_URLUtil.toASCII_ on the url it is given, reencoding already percent-encoded 

For instance, let's say we want to query _http://example.com_, passing a GET 
parameter with name 'q' and value '/'. '/' is a special character, and thus has 
to be encoded before being sent.
If we pass '[_'|] to the 
parserchecker, then it doesn't encode the '/', and tries to fetch the URL as 
is, which is invalid.
If we try to encode the parameter beforehand, and call the parsechecker with 
'', then it encodes the '%' sign to '%25', and thus 
fetches '['.|]

This actually makes it impossible to fetch the correct URL 
( from the parsechecker.

This message was sent by Atlassian JIRA

Reply via email to