rzo1 commented on PR #1984: URL: https://github.com/apache/stormcrawler/pull/1984#issuecomment-4928481908
Thanks for the PR. Aside from the docs thing already mentioned, I have two things while skimming through the PR: 1. HostBackoff imports com.github.benmanes.caffeine.cache.*, but external/urlfrontier/pom.xml doesn't declare caffeine as dependency it; it only compiles via the transitive compile-scope dep from stormcrawler-core. Add com.github.ben-manes.caffeine:caffeine to the urlfrontier pom. 2. backoff.max.secs doesn't cap the first block since the first-incident branch (backoffSecs == 0 ? baseSecs : min(...)) skips the cap, so with max.secs < base.secs the first block uses the full base while later ones are capped lower; a negative max.secs blocks once at base and then never again. I would clamp the first step against maxSecs too (or clamp maxSecs >= baseSecs in the constructor), consistent with the clamping already done for factor, decay and jitter. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
