Author: ab
Date: Mon Jul 24 07:41:18 2006
New Revision: 425071
URL: http://svn.apache.org/viewvc?rev=425071&view=rev
Log:
Expire all finished addresses. When sites request long crawl delays
this quickly ties down all threads, and lock expiration heppens
rarely and proceeds too slowly to remove all expired entries.
Modified:
lucene/nutch/trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
Modified:
lucene/nutch/trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
URL:
http://svn.apache.org/viewvc/lucene/nutch/trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java?rev=425071&r1=425070&r2=425071&view=diff
==============================================================================
---
lucene/nutch/trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
(original)
+++
lucene/nutch/trunk/src/plugin/lib-http/src/java/org/apache/nutch/protocol/http/api/HttpBase.java
Mon Jul 24 07:41:18 2006
@@ -315,8 +315,8 @@
host = url.getHost();
if (host == null)
throw new HttpException("Unknown host for url: " + url);
- host = host.toLowerCase();
}
+ host = host.toLowerCase();
int delays = 0;
while (true) {
@@ -383,8 +383,6 @@
if (time <= System.currentTimeMillis()) {
BLOCKED_ADDR_TO_TIME.remove(host);
BLOCKED_ADDR_QUEUE.removeLast();
- } else {
- break;
}
}
}
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Nutch-cvs mailing list
Nutch-cvs@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nutch-cvs
