rzo1 opened a new issue, #2166: URL: https://github.com/apache/stormcrawler/issues/2166
## What is missing `SiteMapParserBolt` emits one DISCOVERED tuple per URL found in a sitemap, at `core/src/main/java/org/apache/stormcrawler/bolt/SiteMapParserBolt.java:203-206`, with no upper bound. `JSoupParserBolt` has had `parser.emitOutlinks.max.per.page` for the equivalent case since a while, read at `JSoupParserBolt.java:169` and applied at `:604`, but there is no counterpart for sitemaps. ## Why it matters A single sitemap can enrol an arbitrary number of URLs in one tuple. crawler-commons applies its `MAX_URLS = 50000` only to plain-text sitemaps (`SiteMapParser.java:528`); the XML path has no cap at all, which is crawler-commons/crawler-commons#611. Until that is settled upstream, a gzipped sitemap is a cheap way for a site to push tens of thousands of URLs into the status stream from one fetch. A cap here is only a partial mitigation and should be described as one: the parse has already allocated the `SiteMapURL` objects by the time the bolt emits, so this bounds what reaches the downstream queues and the status index, not the parser bolt's own heap. That is still the part an operator can act on. ## Suggested shape A `sitemap.emitOutlinks.max` key, default -1 (no limit) to keep current behaviour, applied in the emit loop and logged when it trims, following what `JSoupParserBolt` already does. Worth deciding whether the truncated sitemap is still marked FETCHED, which is what happens today. Related: #2100 for the unbounded expansion itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
