But then i fetch the other domains www.sf.net <http://www.sf.net/> .....
the output is only
060109 014715 http.agent = NutchCVS/0.7.1 (Nutch;
http://lucene.apache.org/nutch/bot.html; [email protected])
060109 014715 fetcher.server.delay = 5000
060109 014715 http.max.delays = 52
060109 014718 Using URL normalizer: org.apache.nutch.net.BasicUrlNormalizer
060109 014724 status: segment 20060109014654, 3 pages, 0 errors, 51033
bytes, 8309 ms
060109 014724 status: 0.36105427 pages/s, 47.98355 kb/s, 17011.0 bytes/page
there is not output like
060109 154712 fetching http://www.niap.no/magasinet/layout/set/print
060109 154712 fetching http://www.niap.no/magasinet/kontakt_oss
060109 154712 fetching http://www.niap.no/magasinet/ezinfo/about
060109 154712 fetching
http://www.niap.no/index.php/magasinet/nyheter/midt_sten
Stefan Groschupf wrote:
What is java.net.SocketTimeoutException?
Can not connect to the server.
In general you hammer your webserver and it may block the ip of your
server.
You can setup how many threads per host are loading from one host
server.
For a intranet crawl it is a good idea to have less less thread (may
just as much you plan to use at the same time for the host) e.g.
fetcherThreads = 2 maxThreadsPerHost = 2
If you have more threads you should increase the retry / delay
configuration since in case a host is busy with the maximal threads
per host the thread is delayed.
If a thread is delayed to often than you get a Exceeded
http.max.delays: retry later....
Sometimes I'm asking myself if not a queue based fetching would be
better the actually implementation, however this is difficult to change.
HTH
Stefan
------------------------------------------------------------------------
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.1.371 / Virus Database: 267.13.13/195 - Release Date: 08.12.2005
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
