GGraziadei commented on issue #2185: URL: https://github.com/apache/stormcrawler/issues/2185#issuecomment-5815009294
Hi @jnioche thanks a lot for your comment. Happy to discuss here. You're right that it isn't pure FIFO. Inside the selected queue, it pulls URLs from a heap ordered by nextFetchDate. Thanks for pointing me to StatusMetricsBolt. I'm moving the metrics code into a separate bolt and dropping my original approach. My original idea was to improve the queue selection. The linked PR exposes the metrics needed to build a histogram of crawl depths explored so far. With that histogram, queue selection could be tuned to follow a specific crawling strategy. This was the initial strategy, but based on your comment, I’m open to changing it to improve the outcome and make it available to other crawlers as well: - Expose the metrics needed to build the depth histogram (this PR) - Introduce an abstract CrawlingStrategy class - Change the queue key from (host) to (host, depth) (*) - When pulling, pick the depth as well as the host, based on the strategy (*) keying queues on (host, depth) would split a host across several queues, while URLFrontier currently applies politeness per queue. what do you think about? A strategy is defined by a target depth distribution (CDF). Below are a couple of example strategies just to explain my idea (just a sketch generated using Claude to transmit the qualitative idea): <img width="1650" height="1110" alt="Image" src="https://github.com/user-attachments/assets/60c11304-cdf8-4bd9-9782-fece6e222054"; /> --- As per your previous comment, this seems closer to the crawler-commons/url-frontier scope. If you think the idea is worth pursuing, would it be OK if I opened an issue there to discuss the design? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
