[
https://issues.apache.org/jira/browse/NUTCH-2456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244749#comment-16244749
]
Sebastian Nagel commented on NUTCH-2456:
----------------------------------------
Thanks, I've forgot about NUTCH-2184.
{quote}How are these issues really different?{quote}
This issue is included in NUTCH-2184 - if the CrawlDb is not used as all, there
will be definitely no dbDatum available in IndexerMapReduce. NUTCH-2184
includes also an improved command-line option processing and an upgrade of the
MapReduce API (overlaps with NUTCH-2375). As NUTCH-2184 resp. PR
[#95|https://github.com/apache/nutch/pull/95] cannot be merged with the recent
master and there are still a few TODOs in PR #95: why not first commit this
issue?
> Allow to index pages/URLs not contained in CrawlDb
> --------------------------------------------------
>
> Key: NUTCH-2456
> URL: https://issues.apache.org/jira/browse/NUTCH-2456
> Project: Nutch
> Issue Type: Bug
> Components: indexer
> Affects Versions: 1.13
> Reporter: Yossi Tamari
> Priority: Critical
>
> If http.redirect.max is set to a positive value, the Fetcher will follow
> redirects, creating a new CrawlDatum.
> If the redirected URL is fetched and parsed, during indexing for it we have a
> special case: dbDatum is null. This means that in
> [https://github.com/apache/nutch/blob/6199492f5e1e8811022257c88dbf63f1e1c739d0/src/java/org/apache/nutch/indexer/IndexerMapReduce.java#L259]
> the document is not indexed, as it is assumed it only has inlinks (actually
> it has everything but dbDatum).
> I'm not sure what the correct fix is here. It seems to me the condition
> should use AND instead of OR anyway, but I may not understand the original
> intent. It is clear that it is too strict as is.
> However, the code following that line assumes all 4 objects are not null, so
> a patch would need to change more than just the condition.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
