Many thanks, Yossi!
I was beginning to think the problems were along those lines (truncating
long XML files and needing to run bin/nutch sitemap first). Thanks for
clarifying how to fix the issues. I'm successfully running a crawl now.
Chris
On 2018-05-24 07:19 AM, Yossi Tamari wrote:
Hi Chris,
In order to inject sitemaps, you should use the "nutch sitemap" command. After
you inject those sitemaps to the crawl DB, you can proceed as normal with the crawl
command, without the -s parameter.
The error you are seeing may be because you have http.content.limit defined.
The default value would cause any document to be truncated after 65536 bytes.
For sitemaps, you should set it to a much larger number, or -1.
Yossi.
-----Original Message-----
From: Chris Gray <[email protected]>
Sent: 23 May 2018 20:39
To: [email protected]
Subject: Problems starting crawl from sitemaps
I've been using nutch for a few years to do conventional link-to-link crawls of
our local websites, but I would like to switch to doing crawls based on
sitemaps. So far I've had no luck doing this.
I'm not sure I've configured this correctly and the documentation I've found has
left me guessing at many things. Why aren't the pages in listed in a sitemap
being fetched and indexed?
I've installed Nutch 1.14 and Solr 6.6.0. My urls/seeds.txt file contains only
the
URLs for the 4 sitemaps I'm interested in. After running:
bin/crawl -i -D "solr.server.url=http://localhost:8983/solr/nutch"; -s urls
crawl 5
the crawl ends after 3 of 5 iterations and only 3 documents are in the
index: 3 of the seeds.
I do get error messages that 3 sitemap files that contain <urlset> elements are
malformed, for example:
2018-05-23 08:57:24,564 ERROR tika.TikaParser - Error parsing
https://uwaterloo.ca/library/sitemap.xml
Caused by: org.xml.sax.SAXParseException; lineNumber: 420; columnNumber:
122; XML document structures must start and end within the same entity.
But I can't find anything wrong with the sitemaps and other validators say
they're OK and the location pointed to (line 420, column 122) is in the middle
of
the name of a directory in a URL.
Is there good documentation or a tutorial on using Nutch with sitemaps?
