dpol1 opened a new issue, #1979: URL: https://github.com/apache/stormcrawler/issues/1979
Third part of #867, after the Retry-After handling (#784, done in #1973) and the adaptive back-off for headerless rate limits (#1106). Today a robots.txt `Crawl-delay` above `fetcher.max.crawl.delay` has no good outcome: by default the URL is marked `Status.ERROR` (`error.cause=crawl_delay`) and the host is never crawled, with `fetcher.max.crawl.delay.force` it is fetched at a pace that violates what robots asked. Short delays are fine where they are, inside the fetcher queues; it's the long tail that loses either way. And these sites are exactly the ones that mean it - crawl-delay is the most-complied-with robots directive in the wild (https://arxiv.org/abs/2505.21733). The frontier can do better. With `force` enabled the fetcher would report the requested delay in the status metadata, and the queue-stream consumer (`QueueRegulatorBolt`) would pass it to URLFrontier via [setDelay](https://github.com/crawler-commons/url-frontier/blob/master/API/urlfrontier.proto#L55): the host becomes crawlable at the pace robots asked for, and the compliance that `force` alone breaks is restored at the source. Same split as Nutch-style crawlers moved to long ago: politeness state lives with the queue owner, not the workers. Two things worth calling out. Unlike `blockQueueUntil`, `setDelay` does not expire on its own, and `setDelay(key, 0)` erases a server-side default for that queue - so when a site later removes its `Crawl-delay`, the old pace simply stays (throughput lost on that host, never impoliteness) rather than guessing at a reset. And the value needs a cap of its own: SC disables the parser-side limit on crawl-delays, and sites do ask for absurd numbers. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
