lewismc commented on code in PR #820:
URL: https://github.com/apache/nutch/pull/820#discussion_r1759821839
##########
src/java/org/apache/nutch/fetcher/Fetcher.java:
##########
@@ -419,27 +419,43 @@ else if (bandwidthTargetCheckCounter ==
bandwidthTargetCheckEveryNSecs) {
.increment(hitByTimeLimit);
}
- // some requests seem to hang, despite all intentions
+ /*
+ * Some requests seem to hang, with no fetches finished and no new
+ * fetches started during half of the MapReduce task timeout
+ * (mapreduce.task.timeout, default value: 15 minutes). In order to
+ * avoid that the task timeout is hit and the fetcher job is failed,
+ * we stop the fetching now.
+ */
if ((System.currentTimeMillis() - lastRequestStart.get()) > timeout)
{
- if (LOG.isWarnEnabled()) {
- LOG.warn("Aborting with {} hung threads.", activeThreads);
- for (int i = 0; i < fetcherThreads.size(); i++) {
- FetcherThread thread = fetcherThreads.get(i);
- if (thread.isAlive()) {
- LOG.warn("Thread #{} hung while processing {}", i,
- thread.getReprUrl());
- if (LOG.isDebugEnabled()) {
- StackTraceElement[] stack = thread.getStackTrace();
- StringBuilder sb = new StringBuilder();
- sb.append("Stack of thread #").append(i).append(":\n");
- for (StackTraceElement s : stack) {
- sb.append(s.toString()).append('\n');
- }
- LOG.debug(sb.toString());
+ LOG.warn("Aborting with {} hung threads.", activeThreads);
+ innerContext.getCounter("FetcherStatus", "hungThreads")
+ .increment(activeThreads.get());
+ for (int i = 0; i < fetcherThreads.size(); i++) {
+ FetcherThread thread = fetcherThreads.get(i);
+ if (thread.isAlive()) {
+ LOG.warn("Thread #{} hung while processing {}", i,
Review Comment:
Yes I think WARN logging here is called for. That sounds like a better
solution. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
