dpol1 opened a new pull request, #1984: URL: https://github.com/apache/stormcrawler/pull/1984
Stage 2 of #867, closes #1106. #1973 handled the explicit case, a 429/503 carrying a Retry-After header. This covers the one we probably see most in the wild: servers that rate-limit without sending any header. Today we just fetch the host again and collect another 429. A headerless 429/503 arriving on the queue stream now blocks the host's queue in URLFrontier for a growing duration: 60s the first time, doubled per new incident, capped at a day, forgiven after 30 quiet minutes. Same curve as Nutch since 1.19 ([NUTCH-2946](https://issues.apache.org/jira/browse/NUTCH-2946)), but the state lives with the queue owner: a blocked host never reaches the fetcher at all, so the stale fetching Nutch had to patch around ([NUTCH-3114](https://issues.apache.org/jira/browse/NUTCH-3114)) can't happen here. Everything from the #867 discussion is in: * a random fraction on every computed block (`urlfrontier.backoff.jitter`, default 0.1) so hosts blocked on the same schedule don't all come back at once. Thanks @sebastian-nagel, the daily error spike was exactly the failure mode I had dismissed. * configurable trigger codes (`urlfrontier.backoff.status.codes`, default 429 and 503; 403 stays out since it so often just means "forbidden"). They now gate the Retry-After path of #1973 as well, with one behaviour change: a malformed or zero Retry-After escalates as a headerless incident instead of being ignored. * fetch exceptions as incidents behind `urlfrontier.backoff.on.exceptions`, off by default: a timeout is a noisier signal than a clean 429, I'd flip the default once this has seen some real crawls. * URLs already in the fetcher's internal queues just fail and reschedule. Failures during an active block count as the same incident, so a draining backlog escalates at most once per window instead of jumping to the cap. One rename against what I posted on #867: the config prefix is `urlfrontier.backoff.*`, not `fetcher.backoff.*`. The fetcher plays no part in this, and it sits next to Phase 1's `urlfrontier.max.retry.after`. How it's built: * the decision logic is a package-private `HostBackoff`, same split as `RetryAfterParser` in #1973; the bolt keeps the RPC and nothing else. * state is one Caffeine cache with per-entry expiry at block end + decay: memory stays bounded and the 24h cap stays reachable through escalation. * a block is only ever extended, never shrunk by a later, shorter signal. A signal suppressed by that guard or by the anti-burst one re-asserts the stored block with an idempotent RPC, so a lost fire-and-forget call heals on the next signal instead of leaving a phantom block. * `HostBlockBolt` becomes `QueueRegulatorBolt` in a separate mechanical commit: with #1979 lined up next, blocking is one of the actions this bolt takes on a queue, not the only one. * same `metadata.persist` caveat as #1973 (`fetch.statusCode`, plus `fetch.exception` when the flag is on); the bolt warns at startup when they wouldn't survive the filter. Tests: escalation math against a controllable ticker and a seeded RNG for the jitter, regressions for the burst and only-extend guards, and two Testcontainers runs against a real frontier (explicit Retry-After and headerless 429). ### For all changes - [x] Is there a issue associated with this PR? Is it referenced in the commit message? - [ ] Does your PR title start with `#XXXX` where `XXXX` is the issue number you are tryin - [x] Has your PR been rebased against the latest commit within the target branch (typically main)? - [ ] Is your initial contribution a single, squashed commit? (mechanical rename kept as idiff stays reviewable) - [x] Is the code properly formatted with `mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false`? ### For code changes - [x] Have you ensured that the full suite of tests is executed via `mvn clean verify`? - [x] Have you written or updated unit tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? (no new dependencies) - [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file? (n/a) - [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file? (n/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
