dpol1 commented on code in PR #2123:
URL: https://github.com/apache/stormcrawler/pull/2123#discussion_r3992601878
##########
core/src/main/java/org/apache/stormcrawler/persistence/AbstractQueryingSpout.java:
##########
@@ -223,6 +266,27 @@ public void nextTuple() {
timeLastQuerySent = System.currentTimeMillis();
}
+ /** Checks the scheme of a stored URL against the configured {@code
protocols} list. */
+ protected boolean schemeAllowed(String url) {
+ int colon = url.indexOf(':');
+ if (colon <= 0) {
+ return false;
+ }
+ return allowedSchemes.contains(url.substring(0,
colon).toLowerCase(Locale.ROOT));
+ }
+
+ /**
+ * Emits a tuple to the status stream so that the status updater processes
the status, e.g.
+ * removes a row whose URL the spout refuses to emit. The stored metadata
is passed on so that
+ * the status updater does not overwrite the row with an empty set.
+ */
+ protected void emitStatus(String url, Metadata metadata, Status status) {
+ if (metadata == null) {
+ metadata = new Metadata();
+ }
+ collector.emit(Constants.StatusStreamName, new Values(url, metadata,
status));
Review Comment:
Would you set `STATUS_ERROR_CAUSE` here too, like the fetcher and the
updater do, so the store says why the row is ERROR?
##########
docs/src/main/asciidoc/internals.adoc:
##########
@@ -37,6 +37,8 @@ The
link:https://github.com/apache/stormcrawler/blob/main/core/src/main/java/org
The difference between FETCH_ERROR and ERROR is that the former is possibly
transient whereas the latter is terminal. The bolt which is in charge of
updating the status (see below) can then decide when and whether to schedule a
new fetch for a URL based on the status value.
+Note that declaring the status stream in a component is only half the story:
the topology must also connect that component to the StatusUpdaterBolt on
`Constants.StatusStreamName`, otherwise its status tuples go nowhere. The
querying spouts use this for URLs they refuse to emit (a scheme not in the
`protocols` list is reported as ERROR so that the status updater removes the
row from the store); the archetype `crawler.flux` files show the required
wiring (a stream from the spout to the status bolt with `streamId: "status"`).
Topologies built by hand must add the same connection when upgrading.
Review Comment:
Small one: the updater marks the row ERROR (and tells the deletion stream)
rather than removing it. Same wording in the three flux comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
