dpol1 commented on issue #2185: URL: https://github.com/apache/stormcrawler/issues/2185#issuecomment-5832243834
+1 on a separate bolt. It can live in core: it only reads the `depth` metadata of the tuples, so it works with any backend. `(host, depth)` as queue key breaks politeness, URLFrontier applies it per queue. Keep host queues and add a max depth to `GetURLs` frontier side; `GetParams` has nothing else to filter on today. For the bolt: keep the two counters, drop the probability API until something uses it; a depth beyond the cap isn't 1.0; seeds carry no `depth` key, so `depth.0` stays empty. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
