[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sebastian Nagel updated NUTCH-2993: --- Component/s: plugin scoring Affects Version/s: 1.19 > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement > Components: plugin, scoring >Affects Versions: 1.19 >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch, NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. > This patch overrides maxDepth for outlinks of URLs matching a configured > pattern. URL not matching the pattern get the default max depth value > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15-1.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch, NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. > This patch overrides maxDepth for outlinks of URLs matching a configured > pattern. URL not matching the pattern get the default max depth value > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993.patch) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. > This patch overrides maxDepth for outlinks of URLs matching a configured > pattern. URL not matching the pattern get the default max depth value > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. > This patch overrides maxDepth for outlinks of URLs matching a configured > pattern. URL not matching the pattern get the default max depth value > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Description: We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch overrides maxDepth for outlinks of URLs matching a configured pattern. URL not matching the pattern get the default max depth value configured. was: We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch skips the depth check if the current URL matches some regular expression. Initially we tried to set a custom maxDepth based on a Pattern match, but this didn't work. The crawler still managed to creep too deep due to having links everywhere. > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. > This patch overrides maxDepth for outlinks of URLs matching a configured > pattern. URL not matching the pattern get the default max depth value > configured. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15-1.patch) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15.patch) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15-1.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Description: We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch skips the depth check if the current URL matches some regular expression. Initially we tried to set a custom maxDepth based on a Pattern match, but this didn't work. The crawler still managed to creep too deep due to having links everywhere. was:We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch allows for a overridden max depth if the current URL matches against a Pattern. If find(), all outlinks are given a new max depth. > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Summary: ScoringDepth plugin to skip depth check based on URL Pattern (was: ScoringDepth plugin to override maxDepth based on URL Pattern) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
