dpol1 commented on code in PR #2125:
URL: https://github.com/apache/stormcrawler/pull/2125#discussion_r3992602290
##########
core/src/main/resources/crawler-default.yaml:
##########
@@ -288,6 +288,23 @@ config:
# filters URLs in sitemaps based on their modified Date (if any)
sitemap.filter.hours.since.modified: -1
+ # whether a document without the isSitemap key is classified as a sitemap
+ # by searching the first bytes of its content for the sitemaps.org
+ # namespace. Off by default: any page carrying the namespace string early
+ # enough would be reclassified as a sitemap and never reach the parser
+ # bolt. When enabled, a content type which rules a sitemap out (a page
+ # served as HTML) stops the sniffing.
+ sitemap.sniffContent: false
+
+ # whether the sitemap parser applies strict URL checking: a sitemap then
+ # only yields URLs below its own host and path (strict URL checking of
+ # crawler-commons, not its namespace check), so a sitemap cannot enrol URLs
+ # on hosts it has nothing to do with. Off by default: a sitemap living at
Review Comment:
Does strict cover sitemap indexes too? crawler-commons applies the host
check to `<urlset>` entries only, and this bolt marks every index `<loc>` as a
sitemap, so an index can still pull in another host's sitemap.
`SiteMapParser.urlIsValid` in the index loop would close it.
##########
core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:
##########
@@ -103,22 +121,25 @@ public void execute(Tuple tuple) {
LOG.debug("Processing {}", url);
- boolean looksLikeSitemap = sniff(content);
- // can force the mimetype as we know it is XML
- if (looksLikeSitemap) {
+ String isSitemap = metadata.getFirstValue(isSitemapKey);
+
+ // only promote an unmarked document when the operator asked for it: a
+ // page deciding how the pipeline treats it must not depend on a string
+ // in its body, and a promoted document also needs a sitemap compatible
+ // content type
+ if (isSitemap == null && sniffContent && sniffsAsSitemap(ct, content))
{
Review Comment:
A redirected sitemap arrives here without `isSitemap` (the key is persisted,
not transferred), so with sniffing off it is stored as a page with
`isSitemap=false` and stays one. Main promoted it by sniffing; `/sitemap.xml`
to `/sitemap_index.xml` is the common case. Could the fetcher pass the key
along on the redirect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
