[
https://issues.apache.org/jira/browse/NUTCH-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409607#comment-16409607
]
Ben Vachon edited comment on NUTCH-1741 at 3/22/18 2:52 PM:
------------------------------------------------------------
Is there a reason that sitemap page handling has to be separate from
non-sitemap page handling?
The way this is working in 2.x right now is that you generate a list of only
sitemaps, then fetch and parse that list of only sitemaps. Then generate a list
of only non-sitemaps and fetch and parse that list.
What prevents us from allowing Nutch to generate a mixed list and fetch and
parse a mixed list but just handle pages with the InjectType.SITEMAP_INJECT
differently when parsing?
edit:
I read that the idea was to give the user more control. Would anyone be opposed
to me adding an alternate option when running the
GeneratorJob/FetcherJob/ParserJob to create and handle mixed lists?
was (Author: bvachon):
Is there a reason that sitemap page handling has to be separate from
non-sitemap page handling?
The way this is working in 2.x right now is that you generate a list of only
sitemaps, then fetch and parse that list of only sitemaps. Then generate a list
of only non-sitemaps and fetch and parse that list.
What prevents us from allowing Nutch to generate a mixed list and fetch and
parse a mixed list but just handle pages with the InjectType.SITEMAP_INJECT
differently when parsing?
> Support of Sitemaps in Nutch 2.x
> --------------------------------
>
> Key: NUTCH-1741
> URL: https://issues.apache.org/jira/browse/NUTCH-1741
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, generator
> Reporter: Alparslan Avcı
> Assignee: Cihad Guzel
> Priority: Major
> Labels: gsoc2015
> Fix For: 2.4
>
> Attachments: NUTCH-1741-v2.patch, NUTCH-1741-v3.patch,
> NUTCH-1741-v4.patch, NUTCH-1741-webpage-avsc.patch, NUTCH-1741.patch,
> NUTCH-1741v5.patch, NUTCH-1741v6.patch, NUTCH-1741v7.patch,
> SitemapCrawlerLifeCycle.pdf, SitemapDevelopmentFor2x.pdf
>
>
> Sitemap support has to be implemented for 2.x branch. It is being discussed
> in NUTCH-1465 for trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
