abhinav-phi commented on code in PR #2123: URL: https://github.com/apache/stormcrawler/pull/2123#discussion_r3996892691
########## docs/src/main/asciidoc/internals.adoc: ########## @@ -37,6 +37,8 @@ The link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org The difference between FETCH_ERROR and ERROR is that the former is possibly transient whereas the latter is terminal. The bolt which is in charge of updating the status (see below) can then decide when and whether to schedule a new fetch for a URL based on the status value. +Note that declaring the status stream in a component is only half the story: the topology must also connect that component to the StatusUpdaterBolt on `Constants.StatusStreamName`, otherwise its status tuples go nowhere. The querying spouts use this for URLs they refuse to emit (a scheme not in the `protocols` list is reported as ERROR so that the status updater removes the row from the store); the archetype `crawler.flux` files show the required wiring (a stream from the spout to the status bolt with `streamId: "status"`). Topologies built by hand must add the same connection when upgrading. Review Comment: Fixed in 218d6cd9 — `internals.adoc` and the flux comments now say the updater marks the row ERROR and notifies the deletion stream, rather than removing it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
