[
https://issues.apache.org/jira/browse/NUTCH-1457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13493565#comment-13493565
]
Ferdy Galema commented on NUTCH-1457:
-------------------------------------
There is a limited description of the Nutch2 DbUpdater here:
http://wiki.apache.org/nutch/Nutch2Crawling#DbUpdate
Basically all outlinks of the table are needed (they are outputted in the
Mapper), so that they will be available as inlinks in the Reducer. They are
needed to correctly update the Score for each row. This may or may not be
important in some cases, but for a correct calculation of Score all
rows+outlinks are needed.
My suggestion for this issue is to move some of the processing of the DbUpdater
job to the Fetcher. Namely the updating of the fetchstatus and fetchtime. Then
it is possible to run the DpUpdater job many times without pushing fetchtime
far into the future. Slighly related, there should be an extra field to
distinct between "when was this item fetched" and "when should it be fetched".
Now both this information is stored in the single field "fetchtime".
> Nutch2 Refactor the update process so that fetched items are only processed
> once
> --------------------------------------------------------------------------------
>
> Key: NUTCH-1457
> URL: https://issues.apache.org/jira/browse/NUTCH-1457
> Project: Nutch
> Issue Type: Improvement
> Reporter: Ferdy Galema
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
