GGraziadei commented on issue #2303: URL: https://github.com/apache/stormcrawler/issues/2303#issuecomment-6692028161
Thanks @dpol1. I ran your test against current main, and it fails as you described. The underlying issue predates this, but #2132 made it worse by reaping the queue as soon as it drains. One potential fix that reuses the existing semantic is to issue a ticket for nextFetchTime when a drained queue still has a future `nextFetchTime`, rather than reaping it immediately. That way, getFetchItem can reap the queue once the delay has elapsed, while any URL arriving in the meantime still respects the delay. I haven't tried this approach yet, happy to help with your PR review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
