[
https://issues.apache.org/jira/browse/NUTCH-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alparslan Avcı updated NUTCH-1741:
----------------------------------
Attachment: NUTCH-1741.patch
I have uploaded a patch that implements the sitemap support as similar to the
description in [^SitemapDevelopmentFor2x.pdf]. The sitemap crawl cycle consist
of 4 steps:
# _SitemapInjectorJob_ -> This job can inject sitemap urls from robots files of
hosts in _HostDb_ or from a seed list.
# _GeneratorJob_ -> Standard _GeneratorJob_. It has to be run with _-sitemap_
argument.
# _FetcherJob_ -> Standard _FetcherJob_. (I suggest to set _http.content.limit_
(and/or _file.content.limit_) property to _52428800_, since sitemaps can be
large as 50MB)
# _SitemapParserJob_ -> This job parses the sitemaps with specified parse
plugins. As now, I have implemented only one plugin (_parse-sitemap_) that uses
_crawler-commons_ as parser. After parse, the job writes the new urls to
webpage table.
Waiting your comments! :)
> Support of Sitemaps in Nutch 2.x
> --------------------------------
>
> Key: NUTCH-1741
> URL: https://issues.apache.org/jira/browse/NUTCH-1741
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher, generator
> Reporter: Alparslan Avcı
> Fix For: 2.3
>
> Attachments: NUTCH-1741.patch, SitemapDevelopmentFor2x.pdf
>
>
> Sitemap support has to be implemented for 2.x branch. It is being discussed
> in NUTCH-1465 for trunk.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
