[jira] [Updated] (NUTCH-2994) Implement an indexer for OpenSearch 2.x
[ https://issues.apache.org/jira/browse/NUTCH-2994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tim Allison updated NUTCH-2994: --- Description: Over on NUTCH-2920, we added an indexer for OpenSearch 1.x. We should do this for 2.x. This is blocked by: https://github.com/opensearch-project/opensearch-java/issues/181 We could reinvent BulkProcessor/BulkIngester ourselves, but we really, really shouldn't do that. was: Over on NUTCH-2920, we added an indexer for OpenSearch 1.x. We should do this for 2.x. The current blocker is: https://github.com/opensearch-project/opensearch-java/issues/181 We could reinvent BulkProcessor/BulkIngester ourselves, but we really, really shouldn't do that. > Implement an indexer for OpenSearch 2.x > --- > > Key: NUTCH-2994 > URL: https://issues.apache.org/jira/browse/NUTCH-2994 > Project: Nutch > Issue Type: Improvement >Reporter: Tim Allison >Priority: Major > > Over on NUTCH-2920, we added an indexer for OpenSearch 1.x. We should do > this for 2.x. This is blocked by: > https://github.com/opensearch-project/opensearch-java/issues/181 > We could reinvent BulkProcessor/BulkIngester ourselves, but we really, really > shouldn't do that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NUTCH-2994) Implement an indexer for OpenSearch 2.x
Tim Allison created NUTCH-2994: -- Summary: Implement an indexer for OpenSearch 2.x Key: NUTCH-2994 URL: https://issues.apache.org/jira/browse/NUTCH-2994 Project: Nutch Issue Type: Improvement Reporter: Tim Allison Over on NUTCH-2920, we added an indexer for OpenSearch 1.x. We should do this for 2.x. The current blocker is: https://github.com/opensearch-project/opensearch-java/issues/181 We could reinvent BulkProcessor/BulkIngester ourselves, but we really, really shouldn't do that. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15-1.patch) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15.patch) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15-1.patch > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15-1.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Description: We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch skips the depth check if the current URL matches some regular expression. Initially we tried to set a custom maxDepth based on a Pattern match, but this didn't work. The crawler still managed to creep too deep due to having links everywhere. was:We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch allows for a overridden max depth if the current URL matches against a Pattern. If find(), all outlinks are given a new max depth. > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch skips the depth check if the current URL > matches some regular expression. > > Initially we tried to set a custom maxDepth based on a Pattern match, but > this didn't work. The crawler still managed to creep too deep due to having > links everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to skip depth check based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Summary: ScoringDepth plugin to skip depth check based on URL Pattern (was: ScoringDepth plugin to override maxDepth based on URL Pattern) > ScoringDepth plugin to skip depth check based on URL Pattern > > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15.patch > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15.patch) > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: (was: NUTCH-2993-1.15.patch) > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15.patch > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730535#comment-17730535 ] Markus Jelsma commented on NUTCH-2993: -- Here's a simple patch against Nutch 1.15. Will patch for master later. > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
[ https://issues.apache.org/jira/browse/NUTCH-2993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma updated NUTCH-2993: - Attachment: NUTCH-2993-1.15.patch > ScoringDepth plugin to override maxDepth based on URL Pattern > - > > Key: NUTCH-2993 > URL: https://issues.apache.org/jira/browse/NUTCH-2993 > Project: Nutch > Issue Type: Improvement >Reporter: Markus Jelsma >Assignee: Markus Jelsma >Priority: Minor > Fix For: 1.20 > > Attachments: NUTCH-2993-1.15.patch > > > We do not want some crawl to go deep and broad, but instead focus it on a > narrow section of sites. This patch allows for a overridden max depth if the > current URL matches against a Pattern. If find(), all outlinks are given a > new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NUTCH-2993) ScoringDepth plugin to override maxDepth based on URL Pattern
Markus Jelsma created NUTCH-2993: Summary: ScoringDepth plugin to override maxDepth based on URL Pattern Key: NUTCH-2993 URL: https://issues.apache.org/jira/browse/NUTCH-2993 Project: Nutch Issue Type: Improvement Reporter: Markus Jelsma Assignee: Markus Jelsma Fix For: 1.20 We do not want some crawl to go deep and broad, but instead focus it on a narrow section of sites. This patch allows for a overridden max depth if the current URL matches against a Pattern. If find(), all outlinks are given a new max depth. -- This message was sent by Atlassian Jira (v8.20.10#820010)
