GGraziadei opened a new issue, #2129: URL: https://github.com/apache/stormcrawler/issues/2129
FetchItemQueues in FetcherBolt uses a single monitor for `addFetchItem()`, `getFetchItem()` and `finishFetchItem()`, and `getFetchItem()` rotates a `LinkedHashMap` linearly over every queue whose crawl delay has not elapsed. With many hosts the fetcher threads hold that lock most of the time and the executor thread calling `execute()` stalls on every incoming tuple. Benchmark (50 fetcher threads, 1 producer thread, 20 URLs per host, `fetcher.server.delay` 1s, 20000 hosts): ``` getFetchItem avg 1.58 ms, p99 5.0 ms addFetchItem avg 14.1 ms, p99 458 ms, max 571 ms ``` Proposal: keep the queues in a `ConcurrentHashMap` and reference the ones that may be ready from a `DelayQueue` ordered by next fetch time, so that taking an item is O(log n) and adding never waits for the fetcher threads. Same benchmark after the change: `getFetchItem` avg 24 us, `addFetchItem` p99 0.11 ms, throughput unchanged (bounded by politeness). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
