[jira] [Commented] (NUTCH-1643) Unnecessary fetching with http.content.limit when using protocol-http

[Talat UYARER (JIRA)]

    [ 
https://issues.apache.org/jira/browse/NUTCH-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13808006#comment-13808006
 ] 

Talat UYARER commented on NUTCH-1643:
-------------------------------------

Hi [~lewismc], I can look every protocol for this improvement. But we should 
consider your second item. I look at ParseUtil.java  when parsers return null, 
it writes warning log. I think this is not a problem. What do you think ? I 
will uplaod for other protocols. I added parse methods from ParseUtil.java: 

{code:title=ParseUtil.java|borderStyle=solid}
  public Parse parse(String url, WebPage page) throws ParserNotFound, 
      ParseException {
    Parser[] parsers = null;

    String contentType = TableUtil.toString(page.getContentType());

    parsers = this.parserFactory.getParsers(contentType, url);

    for (int i=0; i<parsers.length; i++) {
      if (LOG.isDebugEnabled()) {
        LOG.debug("Parsing [" + url + "] with [" + parsers[i] + "]");
      }
      Parse parse = null;
      
      if (maxParseTime!=-1)
          parse = runParser(parsers[i], url, page);
      else 
          parse = parsers[i].getParse(url, page);
      
      if (parse!=null && ParseStatusUtils.isSuccess(parse.getParseStatus())) {
        return parse;
      }
    }

    LOG.warn("Unable to successfully parse content " + url +
        " of type " + contentType);
    return ParseStatusUtils.getEmptyParse(new ParseException("Unable to 
successfully parse content"), null);
  }
{code}

> Unnecessary fetching with http.content.limit when using protocol-http
> ---------------------------------------------------------------------
>
>                 Key: NUTCH-1643
>                 URL: https://issues.apache.org/jira/browse/NUTCH-1643
>             Project: Nutch
>          Issue Type: Bug
>          Components: protocol
>    Affects Versions: 2.1, 2.2, 2.2.1
>            Reporter: Talat UYARER
>            Priority: Minor
>             Fix For: 2.3
>
>         Attachments: NUTCH-1643.patch, NUTCH-1643v2.patch
>
>
> In protocol-http, Even If I have http.content.limit value set, protocol-http 
> fetches files of all sizes (larger files are fetched until limit allows). 
> But when Parsing, parser skips incomplete files (if parser.skip.truncated 
> configuration is true). It seems like an unnecessary effort to partially 
> fetch contents larger than limit if they are not gonna be parsed.



--
This message was sent by Atlassian JIRA
(v6.1#6144)