abhinav-phi opened a new pull request, #2123: URL: https://github.com/apache/stormcrawler/pull/2123
Fixes #2085. Whatever ends up in the store is fetched: no scheme check and no filtering happens between the store and the fetcher, since URL filtering only runs on the discovery path (`StatusEmitterBolt.filterOutlink` and `URLFilterBolt`). A row whose URL uses a scheme the operator never intended to crawl was emitted as long as a protocol implementation was registered for it — combined with the file scheme being registered by default, a single bad row was a local file read rather than a wasted fetch. `AbstractQueryingSpout.nextTuple` now checks the scheme of each buffered URL against the configured `protocols` list before emitting. Rows with other schemes are skipped at WARN with a `skipped.scheme` counter on the existing event counter, and the buffer keeps draining so one bad row does not block the ones behind it. The full URL filter chain is deliberately not run here: that is hot path cost on every tuple and would silently drop rows an operator deliberately put in the store. A rejected row stays in the store and is skipped again on every query, hence the WARN log. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
