[
https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-2993:
---------------------------------
Description:
We do not want some crawl to go deep and broad, but instead focus it on a
narrow section of sites. This patch skips the depth check if the current URL
matches some regular expression.
Initially we tried to set a custom maxDepth based on a Pattern match, but this
didn't work. The crawler still managed to creep too deep due to having links
everywhere.
was:We do not want some crawl to go deep and broad, but instead focus it on a
narrow section of sites. This patch allows for a overridden max depth if the
current URL matches against a Pattern. If find(), all outlinks are given a new
max depth.
> ScoringDepth plugin to skip depth check based on URL Pattern
> ------------------------------------------------------------
>
> Key: NUTCH-2993
> URL: https://issues.apache.org/jira/browse/NUTCH-2993
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.20
>
> Attachments: NUTCH-2993-1.15.patch
>
>
> We do not want some crawl to go deep and broad, but instead focus it on a
> narrow section of sites. This patch skips the depth check if the current URL
> matches some regular expression.
>
> Initially we tried to set a custom maxDepth based on a Pattern match, but
> this didn't work. The crawler still managed to creep too deep due to having
> links everywhere.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
