[
https://issues.apache.org/jira/browse/NUTCH-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16409607#comment-16409607
]
Ben Vachon edited comment on NUTCH-1741 at 3/22/18 2:55 PM:
------------------------------------------------------------
Is there a reason that sitemap page handling has to be separate from
non-sitemap page handling?
The way this is working in 2.x right now is that you generate a batch of only
sitemaps, then fetch and parse that batch of only sitemaps. Then generate a
batch of only non-sitemaps and fetch and parse that batch.
What prevents us from allowing Nutch to generate a mixed batch and fetch and
parse a mixed batch but just handle pages with the InjectType.SITEMAP_INJECT
differently when parsing?
edit:
I read that the idea was to give the user more control. Would anyone be opposed
to me adding an alternate option when running the
GeneratorJob/FetcherJob/ParserJob to create and handle mixed batches (basically
just making the "-sitemaps" argument flag into a ternary operator)?
was (Author: bvachon):
Is there a reason that sitemap page handling has to be separate from
non-sitemap page handling?
The way this is working in 2.x right now is that you generate a batch of only
sitemaps, then fetch and parse that batch of only sitemaps. Then generate a
batch of only non-sitemaps and fetch and parse that batch.
What prevents us from allowing Nutch to generate a mixed batch and fetch and
parse a mixed batch but just handle pages with the InjectType.SITEMAP_INJECT
differently when parsing?
edit:
I read that the idea was to give the user more control. Would anyone be opposed
to me adding an alternate option when running the
GeneratorJob/FetcherJob/ParserJob to create and handle mixed batches?
> Support of Sitemaps in Nutch 2.x
> --------------------------------
>
> Key: NUTCH-1741
> URL: https://issues.apache.org/jira/browse/NUTCH-1741
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, generator
> Reporter: Alparslan Avcı
> Assignee: Cihad Guzel
> Priority: Major
> Labels: gsoc2015
> Fix For: 2.4
>
> Attachments: NUTCH-1741-v2.patch, NUTCH-1741-v3.patch,
> NUTCH-1741-v4.patch, NUTCH-1741-webpage-avsc.patch, NUTCH-1741.patch,
> NUTCH-1741v5.patch, NUTCH-1741v6.patch, NUTCH-1741v7.patch,
> SitemapCrawlerLifeCycle.pdf, SitemapDevelopmentFor2x.pdf
>
>
> Sitemap support has to be implemented for 2.x branch. It is being discussed
> in NUTCH-1465 for trunk.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
