Hi folks, I have posted a first release candidate for the Apache StormCrawler 3.7.0 release and it is ready for testing.
This release introduces a queue stream for the URLFrontier module, used to back off from rate-limited hosts, hold queues on Retry-After and enforce robots.txt Crawl-delay directives. The Selenium protocol, the CloudSearch indexer and the Apache HttpClient-based protocol are deprecated and will be removed in the next major release. It also upgrades to Apache Storm 2.8.9. Note that 3.7.0 is planned to be the final release of the 3.x line, as Apache Storm 2.x has reached end of life. The website changes for this release are on the release-3.7.0 branch of stormcrawler-site [1] - feedback is welcome, in particular on whether we should state the 3.x EOL explicitly on the site (e.g. on the download page) once 4.0 is out. Thank you to everyone who contributed to this release, including all of our users and the people who submitted bug reports, contributed code or documentation enhancements. [1] https://github.com/apache/stormcrawler-site/tree/release-3.7.0 The release was made using the Apache StormCrawler release process, documented here: https://github.com/apache/stormcrawler/blob/main/RELEASING.md Source: https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.7.0-rc1 Tag: https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.7.0 Commit Hash: 604a97328f83073b98f1b707b38293d832fe2b81 Maven Repo: https://repository.apache.org/content/repositories/orgapachestormcrawler-1038 <repositories> <repository> <id>stormcrawler-3.7.0-rc1</id> <name>Testing StormCrawler 3.7.0 release candidate</name> <url> https://repository.apache.org/content/repositories/orgapachestormcrawler-1038 </url> </repository> </repositories> Preview of website: https://stormcrawler.staged.apache.org/download/index.html Release notes: https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.7.0 Reminder: The up-2-date KEYS file for signature verification can be found here: https://downloads.apache.org/stormcrawler/KEYS Please vote on releasing these packages as Apache StormCrawler 3.7.0 The vote is open for at least the next 72 hours. Only votes from the StormCrawler PMC are binding, but everyone is welcome to check the release candidate and vote. The vote passes if at least three binding +1 votes are cast. Please VOTE [+1] go ship it [+0] meh, don't care [-1] stop, there is a ${showstopper} Thanks! Davide
