dpol1 commented on code in PR #2135:
URL: https://github.com/apache/stormcrawler/pull/2135#discussion_r3992602690
##########
core/src/main/java/org/apache/stormcrawler/Constants.java:
##########
@@ -60,4 +60,13 @@ public class Constants {
public static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
private Constants() {}
+
+ /** Hard deadline in seconds for a single fetch, -1 to disable. */
+ public static final String FETCH_TIMEOUT_PARAM_KEY =
"fetcher.thread.timeout";
Review Comment:
`FetcherBolt.FETCH_TIMEOUT_PARAM_KEY` shipped in 3.6.0 and 3.7.0, so code
that used it stops compiling. Keep a deprecated alias, or note it for 4.0?
##########
core/src/main/java/org/apache/stormcrawler/bolt/FetcherBolt.java:
##########
@@ -1002,10 +954,19 @@ public void run() {
}
// common exceptions for which we log only a short message
- if (exece.getCause() instanceof
java.util.concurrent.TimeoutException
+ if (exece instanceof FetchTimeoutHelpers.TimeoutException
+ || exece instanceof java.io.InterruptedIOException
|| message.contains(" timed out")) {
LOG.info("Socket timeout fetching {}", fit.url);
message = "Socket timeout fetching";
+ eventCounter.scope("fetch.timeout").incrBy(1);
+ } else if (exece instanceof
FetchTimeoutHelpers.SaturatedException) {
+ eventCounter.scope("fetch.helper.rejected").incrBy(1);
+ LOG.warn(
+ "{}: all {} fetch helpers are busy",
+ message,
+ fetchHelpers.maxHelpers());
+ message = "No fetch helper available";
Review Comment:
This goes out as FETCH_ERROR, so a URL that never reached the network still
takes a strike towards max.fetch.errors. Would the queue-timeout treatment
above (ack, no status, spout retries) fit better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
