GGraziadei opened a new pull request, #2186: URL: https://github.com/apache/stormcrawler/pull/2186
Fixes #2185 With URLFrontier the order of the crawl is decided by the frontier: rotation between host queues, FIFO within each queue, depth plays no role. This PR adds observability of the depth of what the frontier hands out, without changing the behaviour of the crawl. ## Changes - `Spout` registers a `depth` counter with one scope per depth level read from the `depth` metadata (`depth.0`, `depth.1`, ...), an `unknown` scope for URLs without a numeric depth, and a single `N+` scope from `urlfrontier.depth.metric.max` (default 10) upwards, so that the number of scopes stays bounded when no `MaxDepthFilter` is configured. - A `depth_le` counter with cumulative counts in the style of Prometheus histograms: `depth_le.X` is the number of URLs with depth <= X, `depth_le.inf` the number of URLs with a valid depth, so P(depth <= X) is a plain ratio in a dashboard. - `probabilityDepthAtMost(int)` on the spout returns that ratio in constant time from an in-memory copy of the cumulative counts, since the spout was opened. - `DepthWindow`, a ring of ten time slots covering `urlfrontier.depth.window.secs` (default 300), and `windowProbabilityDepthAtMost(int)` returning the same ratio over the recent URLs only, for a future control loop in `populateBuffer`. - README section documenting the metrics and the two configuration keys. ## Tests - `SpoutDepthMetricTest`: counts per level, `unknown` and `N+` buckets, configurable cap, cumulative counters, probability values, NaN without data, window wiring. - `DepthWindowTest`: recent data, cap, expiry beyond the window, slot reuse after a full turn, NaN when empty. - `mvn -pl external/urlfrontier verify -Dskip.format.code=false`: 91 tests, formatter and Checkstyle clean. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
