dpol1 commented on PR #2000: URL: https://github.com/apache/stormcrawler/pull/2000#issuecomment-4988895259
While reviewing the distributed path, I found that URLFrontier 2.5 does not appear to propagate `setDelay` or `blockQueueUntil` when `local=false`. This is now tracked upstream in crawler-commons/url-frontier#146. I am moving this PR back to draft while I: - make host-level robots pacing conservative with max-observed-wins; - introduce a robots-specific maximum delay; - clarify or enforce the supported deployment model until the upstream behaviour is resolved; - update tests and documentation to reflect the revised semantics. The existing single-node implementation and tests remain valid. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
