[
https://issues.apache.org/jira/browse/NUTCH-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma closed NUTCH-158.
-------------------------------
Resolution: Won't Fix
Bulk close of legacy issues:
http://www.lucidimagination.com/search/document/2738eeb014805854/clean_up_open_legacy_issues_in_jira
> Process Sitemap data in text, rss or xml format as well as OAI-PMH
> ------------------------------------------------------------------
>
> Key: NUTCH-158
> URL: https://issues.apache.org/jira/browse/NUTCH-158
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 0.8
> Reporter: byron miller
> Priority: Minor
>
> Add support to the fetcher to look for sitemap files, download them and
> process them into webdb.
> Perhaps create a robots.txt directive that can be used to create a standard
> format for sitemaps in RSS, XML or text format (one line per url) and process
> that.
> I would love to see someone stomp on proprietary sitemap features or making
> things so google specific as they are today :)
> * RSS format/Atom Format (standard)
> * XML meta descroption
> * OAI-PMH meta description
> (http://www.openarchives.org/OAI/openarchivesprotocol.html)
> Perhaps even a "pre crawler" that will scour for these to inject into the web
> db to help build your link map so you could even just index topN.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
