dpol1 commented on PR #2000: URL: https://github.com/apache/stormcrawler/pull/2000#issuecomment-5073494640
Upstream update: both URLFrontier issues are now fixed on master — crawler-commons/url-frontier#148 routes keyed `setDelay`/`blockQueueUntil` to the queue owner, and crawler-commons/url-frontier#151 / crawler-commons/url-frontier#156 make the `getURLs` politeness gate atomic. No changes needed here: the bolt already sends keyed calls with `local=false`, which is exactly the path the fix routes. Since there is no release containing the fixes yet, the single-endpoint restriction stays; once one is out I plan a follow-up to bump the dependency, add a distributed test and drop the fail-fast. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
