GGraziadei commented on PR #2132: URL: https://github.com/apache/stormcrawler/pull/2132#issuecomment-5590014884
All three observations verified with tests before changing anything, fixed in 5f40940. **Racy max on the crawl delays.** Confirmed. A spin-barrier test with two long-lived threads calling `getFetchItemQueue` for the same host with delays 1000/2000 lost the larger value in 2284 of 300000 rounds on the previous revision (a `CyclicBarrier` version never hit it: thread release skew is orders of magnitude above the window). `crawlDelay` and `minCrawlDelay` are now `AtomicLong`s raised with `accumulateAndGet(v, Math::max)`; the fetcher thread keeps setting them outright from robots.txt as it did before, which was already outside the old lock. Test `concurrentDelayUpdatesKeepTheMaximum`, 300000 rounds in about 0.2 s. **Size decremented after the dequeue.** Confirmed, deterministically: a queue with `maxQueueSize = 1`, poll paused via the `afterDequeue()` hook right after the item is out, a concurrent `addFetchItem` returned false. The size is now reserved before the dequeue and given back if the queue was empty, mirroring `tryAcquireSlot`. The `offer()` javadoc now says why its own transient over-report is harmless (offers are serialised by the monitor) instead of the wrong claim about metrics; `getQueueSize()` clamps at zero for the metrics while a poll of an empty queue is in flight. Test `addDuringPollIsNotRefusedAtTheSizeBound`. **Stale ticket returning null.** Confirmed, deterministically: `a.net` ticket at the head with its delay extended to 10 s by a `finish`, `b.net` due, `getFetchItem` returned null. Now `continue`; the re-issued ticket carries a future time so it cannot come straight back, and the loop is bounded anyway. Test `staleTicketDoesNotHideAnotherReadyQueue`. Same point as @dpol1's inline question. Full `core` verify: 457 tests, 0 failures. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01C6YDC99NSJKU8JeWF2sGfY -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
