[
https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738047#comment-17738047
]
Markus Jelsma commented on NUTCH-2993:
--------------------------------------
Thanks Sebastian!
# changed the checks again.
# check for empty/non-configured pattern in place
# props added to config
# try/catch in place
# typo removed
Ran a few long crawls with just over a hundred domains. I changed the checks
again. Now the maxDepth resets if it does NOT match/find the pattern. There was
still a possibility of sitemap-like pages being passed an overridden maxDepth,
due to a linking page matching the pattern, and then a whole site got crawled
anyway.
> ScoringDepth plugin to skip depth check based on URL Pattern
> ------------------------------------------------------------
>
> Key: NUTCH-2993
> URL: https://issues.apache.org/jira/browse/NUTCH-2993
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Minor
> Fix For: 1.20
>
> Attachments: NUTCH-2993-1.15-1.patch, NUTCH-2993-1.15.patch
>
>
> We do not want some crawl to go deep and broad, but instead focus it on a
> narrow section of sites.
> This patch overrides maxDepth for outlinks of URLs matching a configured
> pattern. URL not matching the pattern get the default max depth value
> configured.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
