Here is my +1 (non-binding) The queue stream needed a controlled setup to be tested properly (you can't politely trigger Retry-After handling against real sites), so I ran the RC against a local server serving a Crawl-delay in robots.txt and endpoints answering 429/503 with and without Retry-After. Queues get paced and blocked as declared, back-off escalates on repeated 429s, and other hosts keep crawling at full speed meanwhile.
Agreed on both nits: I'll open a PR for the NOTICE year, and we'll add the 3.x EOL note to the download page once 4.0 ships. Neither blocks the release. Thanks to everyone who tested the RC! Cheers, Davide Il giorno ven 31 lug 2026 alle ore 18:40 Markos Volikas <[email protected]> ha scritto: > Hi everyone, > > Thanks Davide for preparing the release! > > Here is my +1 (binding): > > - Checked hashes > - Built from source > - Ran a crawl with Solr 10.0.0 in cluster mode and Storm 2.8.9 > (everything on the host / no containers) > > Markos > > On 7/29/26 17:59, Davide Polato wrote: > > Hi folks, > > > > I have posted a first release candidate for the Apache StormCrawler 3.7.0 > > release and it is ready for testing. > > > > This release introduces a queue stream for the URLFrontier module, used > to > > back off from rate-limited hosts, hold queues on Retry-After and enforce > > robots.txt Crawl-delay directives. The Selenium protocol, the CloudSearch > > indexer and the Apache HttpClient-based protocol are deprecated and will > be > > removed in the next major release. It also upgrades to Apache Storm > 2.8.9. > > > > Note that 3.7.0 is planned to be the final release of the 3.x line, as > > Apache Storm 2.x has reached end of life. The website changes for this > > release are on the release-3.7.0 branch of stormcrawler-site [1] - > feedback > > is welcome, in particular on whether we should state the 3.x EOL > explicitly > > on the site (e.g. on the download page) once 4.0 is out. > > > > Thank you to everyone who contributed to this release, including all of > our > > users and the people who submitted bug reports, > > contributed code or documentation enhancements. > > > > [1]https://github.com/apache/stormcrawler-site/tree/release-3.7.0 > > > > The release was made using the Apache StormCrawler release process, > > documented here: > > https://github.com/apache/stormcrawler/blob/main/RELEASING.md > > > > Source: > > > > > https://dist.apache.org/repos/dist/dev/stormcrawler/stormcrawler-3.7.0-rc1 > > > > Tag: > > > > https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.7.0 > > > > Commit Hash: > > > > 604a97328f83073b98f1b707b38293d832fe2b81 > > > > Maven Repo: > > > > > https://repository.apache.org/content/repositories/orgapachestormcrawler-1038 > > > > <repositories> > > <repository> > > <id>stormcrawler-3.7.0-rc1</id> > > <name>Testing StormCrawler 3.7.0 release candidate</name> > > <url> > > > https://repository.apache.org/content/repositories/orgapachestormcrawler-1038 > > </url> > > </repository> > > </repositories> > > > > Preview of website: > > > > https://stormcrawler.staged.apache.org/download/index.html > > > > Release notes: > > > > https://github.com/apache/stormcrawler/releases/tag/stormcrawler-3.7.0 > > > > Reminder: The up-2-date KEYS file for signature verification can be > > found here:https://downloads.apache.org/stormcrawler/KEYS > > > > Please vote on releasing these packages as Apache StormCrawler 3.7.0 > > The vote is open for at least the next 72 hours. > > > > Only votes from the StormCrawler PMC are binding, but everyone is welcome > > to check the release candidate and vote. > > The vote passes if at least three binding +1 votes are cast. > > > > Please VOTE > > > > [+1] go ship it > > [+0] meh, don't care > > [-1] stop, there is a ${showstopper} > > > > Thanks! > > > > Davide > >
