dpol1 opened a new pull request, #2000:
URL: https://github.com/apache/stormcrawler/pull/2000
Closes #1979. Completes Stage 3: the queue-side politeness work discussed in
#867, after explicit `Retry-After` handling in #1973 and adaptive back-off for
headerless rate limits in #1984.
When `fetcher.max.crawl.delay.force` is enabled, the fetcher caps an
excessive robots.txt Crawl-delay locally and continues fetching. Until now,
that meant losing the original interval and potentially crawling the host
faster than requested. Both fetcher bolts now preserve the requested delay in
`robots.crawl.delay`, and `QueueRegulatorBolt` forwards it to URLFrontier
through `setDelay`.
### Design
- The signal is emitted only when the fetcher actually caps a robots.txt
Crawl-delay. Fractional delays are rounded up to whole seconds, so frontier
pacing is never more aggressive than robots.txt.
- The signal is rebuilt from the locally parsed robots rules after
protocol metadata is merged. Stale metadata or a colliding HTTP response header
therefore cannot inject or alter the queue delay.
- Robots pacing is opt-in through
`urlfrontier.robots.crawl.delay.enabled`. `CrawlDelayPolicy` caps values with
`urlfrontier.backoff.max.secs`, keeps one RPC in flight per host, and coalesces
concurrent updates so the latest value eventually wins.
- Successful values are deduplicated for `urlfrontier.backoff.decay.secs`.
A failed RPC retries only a value received while the previous call was in
flight; otherwise a later queue signal reasserts it.
- `setDelay(key, 0)` is deliberately never sent. If a site removes its
Crawl-delay, its queue may remain slower than necessary, but the crawler never
becomes less polite or erases a frontier-side default.
### Configuration contract
The feature fails fast unless the queue and metadata configuration can
enforce a per-host delay safely:
```yaml
partition.url.mode: byHost
fetcher.max.crawl.delay.force: true
urlfrontier.robots.crawl.delay.enabled: true
urlfrontier.max.urls.per.bucket: 1
urlfrontier.backoff.max.secs: 86400
metadata.persist:
- robots.crawl.delay
```
`robots.crawl.delay` must not appear in `metadata.transfer`, including
through wildcards such as `robots.*`: an outlink must not inherit the delay of
its parent host. Custom `metadata.transfer.class` implementations remain
responsible for preserving the same contract.
URLFrontier applies the delay between queue hand-outs; it does not shrink
an existing batch or recall URLs already emitted. Requiring one URL per bucket
bounds each new hand-out, while already prefetched or concurrent requests may
still complete during the transition.
### Tests
Coverage includes both fetcher implementations, fractional delay rounding,
stale and colliding metadata, opt-in and startup validation, asynchronous
last-value-wins serialization, failed and stale callbacks, deduplication, caps,
and a Testcontainers test against a real URLFrontier proving that the second
URL remains unavailable during the configured delay.
The core and URLFrontier module suites pass with 334 and 55 tests
respectively. Configuration, module README, defaults and extension
documentation have been updated.
### For all changes
- [x] Is there a issue associated with this PR? Is it referenced in the
commit message?
- [x] Does your PR title start with `#XXXX` where `XXXX` is the issue
number you are trying to resolve?
- [x] Has your PR been rebased against the latest commit within the target
branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit? (kept as
eight atomic commits so the producer, policy, integration, safety validation
and documentation changes can be reviewed separately)
- [x] Is the code properly formatted with `mvn git-code-format:format-code
-Dgcf.globPattern="**/*" -Dskip.format.code=false`?
### For code changes
- [ ] Have you ensured that the full suite of tests is executed via `mvn
clean verify`? (core and URLFrontier module suites pass; full reactor pending)
- [x] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? (no new
dependencies)
- [ ] If applicable, have you updated the LICENSE file, including the main
LICENSE file? (not applicable)
- [ ] If applicable, have you updated the NOTICE file, including the main
NOTICE file? (not applicable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
