The Apache StormCrawler team is pleased to announce the release of version 3.7.0 of Apache StormCrawler. StormCrawler is a collection of resources for building low-latency, customisable and scalable web crawlers on Apache Storm.
Apache StormCrawler 3.7.0 source distributions is available for download from our download page: https://stormcrawler.apache.org/download/index.html Apache StormCrawler is distributed by Maven Central as well. Changes in this version: - A queue stream for the URLFrontier module, used to back off from rate-limited hosts, hold queues on Retry-After and enforce robots.txt Crawl-delay directives - The Selenium protocol, the CloudSearch indexer and the Apache HttpClient-based protocol are deprecated and will be removed in the next major release - Upgrade to Apache Storm 2.8.9 Please note that 3.7.0 is the final release of the StormCrawler 3.x line, as Apache Storm 2.x has reached end of life. Development continues with StormCrawler 4.0, which will be based on Apache Storm 3. For a complete list of fixed bugs and improvements please see the release notes on GitHub: https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.7.0 The Apache StormCrawler Team
