dpol1 commented on code in PR #2135:
URL: https://github.com/apache/stormcrawler/pull/2135#discussion_r4012838836
##########
core/src/main/java/org/apache/stormcrawler/protocol/okhttp/HttpProtocol.java:
##########
@@ -731,6 +748,16 @@ public ProtocolResponse getProtocolOutput(String url,
final Metadata metadata)
}
call = fetchClient.newCall(currentRequest);
hopCalls.add(call);
+ if (deadlineNanos != 0) {
+ // hard deadline for the whole chain, enforced by okio's
watchdog: on
Review Comment:
DNS can still hold the fetcher thread past this deadline: cancellation
cannot release the synchronous lookup until the resolver returns. Could we say
so next to the hard-deadline description?
##########
core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java:
##########
@@ -1017,17 +993,33 @@ public void run() {
collector.emit(Constants.StatusStreamName, fit.tuple,
tupleToSend);
}
+ } catch (FetchTimeoutHelpers.SaturatedException e) {
+ // the URL never reached the network: like a URL which
waited too long in
+ // the queue, it is acked without a status so that the
spout retries it
+ // later, rather than taking a strike towards
max.fetch.errors
+ eventCounter.scope("fetch.helper.rejected").incrBy(1);
+ LOG.warn(
+ "[Fetcher #{}] {}: all {} fetch helpers are busy",
+ taskId,
+ e.getMessage(),
+ fetchHelpers.maxHelpers());
+ asap = true;
Review Comment:
With asap = true a queue with more URLs is eligible again immediately while
the helper pool is still full, so it collects repeated rejections and WARNs.
Could saturation get some backoff?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
