dpol1 commented on PR #2000: URL: https://github.com/apache/stormcrawler/pull/2000#issuecomment-5022539257
Done in e2cda7af: pacing now coalesces to the max observed delay (a failed RPC keeps its value as a floor for the decay window, so a lower signal can't undercut it), the cap and dedupe window moved to dedicated urlfrontier.robots.delay.* keys, and multiple frontier addresses fail fast only when robots pacing is enabled (just a warning otherwise, matching the #1984 behaviour). While double-checking the concurrent path I also found the getURLs politeness gate itself is racy under concurrent clients — filed as crawler-commons/url-frontier#147 and documented here rather than enforced. PR body updated, marking as ready. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
