lewismc commented on code in PR #820:
URL: https://github.com/apache/nutch/pull/820#discussion_r1757740053
##########
src/java/org/apache/nutch/fetcher/Fetcher.java:
##########
@@ -419,27 +419,43 @@ else if (bandwidthTargetCheckCounter ==
bandwidthTargetCheckEveryNSecs) {
.increment(hitByTimeLimit);
}
- // some requests seem to hang, despite all intentions
+ /*
+ * Some requests seem to hang, with no fetches finished and no new
+ * fetches started during half of the MapReduce task timeout
+ * (mapreduce.task.timeout, default value: 15 minutes). In order to
+ * avoid that the task timeout is hit and the fetcher job is failed,
+ * we stop the fetching now.
+ */
if ((System.currentTimeMillis() - lastRequestStart.get()) > timeout)
{
- if (LOG.isWarnEnabled()) {
- LOG.warn("Aborting with {} hung threads.", activeThreads);
- for (int i = 0; i < fetcherThreads.size(); i++) {
- FetcherThread thread = fetcherThreads.get(i);
- if (thread.isAlive()) {
- LOG.warn("Thread #{} hung while processing {}", i,
- thread.getReprUrl());
- if (LOG.isDebugEnabled()) {
- StackTraceElement[] stack = thread.getStackTrace();
- StringBuilder sb = new StringBuilder();
- sb.append("Stack of thread #").append(i).append(":\n");
- for (StackTraceElement s : stack) {
- sb.append(s.toString()).append('\n');
- }
- LOG.debug(sb.toString());
+ LOG.warn("Aborting with {} hung threads.", activeThreads);
+ innerContext.getCounter("FetcherStatus", "hungThreads")
+ .increment(activeThreads.get());
+ for (int i = 0; i < fetcherThreads.size(); i++) {
+ FetcherThread thread = fetcherThreads.get(i);
+ if (thread.isAlive()) {
+ LOG.warn("Thread #{} hung while processing {}", i,
Review Comment:
I wonder if we could improve this logging by explicitly stating that
DEBUG-level logging can be turned on for Stack-level granularity. Otherwise the
Nutch admin would never know that.
wdyt @sebastian-nagel ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nutch.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
