The GitHub Actions job "Java CI with Maven" on stormcrawler.git/perf/fetch-queues-lock-free has succeeded. Run started by GitHub user GGraziadei (triggered by GGraziadei).
Head commit for run: 47215be18a72786d7bdc487ad94ceb8ee1bdd44b / Gianluca Graziadei <[email protected]> #2129 FetcherBolt: lock-free fetch queues ordered by next fetch time FetchItemQueues used a single monitor for adding, taking and finishing items, and getFetchItem() rotated a LinkedHashMap linearly over all the queues that were not ready yet. With many hosts the fetcher threads held that lock most of the time and the executor thread calling execute() stalled on every incoming tuple. Queues are now kept in a ConcurrentHashMap and the ones that may have an item ready are referenced from a DelayQueue of tickets ordered by their next fetch time, so taking an item is O(log n) and adding never waits for the fetcher threads. Per-queue state (size bound, in-progress count, next fetch time, crawl delays) is handled with atomics and a per-queue monitor only for the add/reap race. Empty queues are removed from the map as soon as they drain. Fixes #2129. Behaviour preserved: politeness per queue, asap release, max threads per queue, max queue size, crawl delay overrides from metadata and robots.txt, queue modes, metrics and the debug dump. Covered by the new FetchItemQueuesTest, including a concurrent producers/consumers test. Benchmark (50 fetcher threads, 1 producer, 20 URLs per host, delay 1s): 20000 hosts getFetchItem avg 1.58 ms -> 24 us addFetchItem p99 458 ms -> 0.11 ms, max 571 ms -> 0.8 ms throughput unchanged (bounded by politeness) Report URL: https://github.com/apache/stormcrawler/actions/runs/34036392979 With regards, GitHub Actions via GitBox
