[
https://issues.apache.org/jira/browse/NUTCH-2261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285086#comment-15285086
]
David Astle commented on NUTCH-2261:
------------------------------------
Repro steps:
- Enable DepthScoring filter (so that depth metadata is added)
- Inject URL with content-level redirect and a max depth metadata value (such
as http://www.mpmo.nl)
- Run generate, fetch, parse, and updatedb
- Run generate, fetch, parse, and update db again
- The metadata in the second segment will have the default _maxdepth_ and
_depth_ values, regardless of what was set for the injected URL
> ParseSegment job does not pass metadata for content-level redirects
> -------------------------------------------------------------------
>
> Key: NUTCH-2261
> URL: https://issues.apache.org/jira/browse/NUTCH-2261
> Project: Nutch
> Issue Type: Bug
> Components: metadata, parser
> Affects Versions: 1.11, 1.12
> Reporter: David Astle
> Priority: Minor
>
> When Fetcher runs in parsing mode, CrawlDatum metadata is properly passed to
> a new CrawlDatum for content-level redirects (HTML meta tag "Refresh"). If
> Fetcher runs in non-parsing mode, and ParseSegment is run as a separate step,
> then metadata other than "_repr_" is not passed to the new CrawlDatum.
> This means that any filter relying on metadata, such as DepthScoringFilter
> and URLMetaScoringFilter, will not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
