GGraziadei commented on PR #2186: URL: https://github.com/apache/stormcrawler/pull/2186#issuecomment-5812951828
I checked the queue selection in url-frontier on `master` (HEAD 43c58c3, 2026-09-22) and on the 2.3.1, 2.5 and 2.6 tags, since the behaviour matters for how this is framed. **Between queues** the frontier does not look at `nextFetchDate`: `getURLs` takes the first queue of the insertion-ordered map and moves it to the tail (`rotateFirstEntry()` on `ConcurrentInsertionOrderMap` since 2.6, `pollFirstEntry()` + `put` in 2.5, `iterator.remove()` + `put` in 2.3.1). The only gate is politeness (`blockedUntil`, `delay`, `lastProduced`); `QueueInterface` has no notion of a date. So the selection is round-robin across queues, in every release. **Within a queue** you are right and my description was imprecise: URLs are served by `nextFetchDate` (RocksDB scheduling key `crawl_queue_paddedDate_url`, the iterator stops at the first future date), not FIFO. Discovered URLs get `Instant.now()` on insertion, so among them it is arrival order, which is why depth ends up correlated with the position in the queue; known URLs with a refetch date interleave by date. I will fix the wording in the issue and in the description. On the rest I agree: steering belongs in url-frontier, and this PR should stay pure observability. On where to observe: `StatusMetricsBolt` polls the store on tick tuples, while the depth has to be read from the tuples the spout emits, so the equivalent is a bolt subscribed to the spout's default stream that reads the `depth` metadata, counts and acks. That keeps the URLFrontier spout untouched, works with any backend (OpenSearch, SQL, ...) and can live in `core`. Unless you prefer otherwise I will rework the PR that way: `DepthMetricsBolt` in `core` with the `depth` / `depth_le` counters and the windowed P(depth <= X) exposed as a gauge, nothing left in the spout. Measuring inside url-frontier would need the frontier to know which metadata key carries the depth (it is a StormCrawler key); I can open an issue on crawler-commons for a configurable key, as a complement rather than a replacement, since a frontier-side metric could not be broken down per topology. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
