Kragen,
Thanks, I like your solution - I hope the other nutch developers will
consider this change into the main branch of code.
Vince
On Fri, 03 Dec 2004 09:35:45 -0800, Kragen Sitaker
<[EMAIL PROTECTED]> wrote:
> On Fri, 2004-12-03 at 00:34 -0700, Vince Taluskie wrote:
>
>
> > What about allowing fetcher.server.delay to be in millisecs? This would
> > also help out our situation where a single host serves up our intranet,
> > but we've got 100,000 documents. The limitation of only servicing 1
> > hit/second is making our crawls take about 2 days and we'd like to go to
> > nightly. When I've tried to run with delay set to '0' I get a hang
> > where from the nutch crawl command where no further output is logged and
> > the java process associated with the nutch crawl command just spins on
> > cpu.
>
> We had very similar problems at CommerceNet. I solved them by parsing
> fetcher.server.delay as a float. As it turns out, the SERVER_DELAY
> constant that the config gets read into is actually already in
> milliseconds.
>
> Index: src/plugin/protocol-http/src/java/net/nutch/protocol/http/Http.java
> ===================================================================
> --- src/plugin/protocol-http/src/java/net/nutch/protocol/http/Http.java
> (revision 4)
> +++ src/plugin/protocol-http/src/java/net/nutch/protocol/http/Http.java
> (revision 5)
> @@ -45,7 +45,7 @@
> static String AGENT_STRING = getAgentString();
>
> static long SERVER_DELAY =
> - NutchConf.getInt("fetcher.server.delay", 1) * 1000;
> + (long)(NutchConf.getFloat("fetcher.server.delay", (float)1.0) * 1000);
>
> static {
> LOG.info("http.proxy.host = " + PROXY_HOST);
>
> HTH. Please pardon my pidgin Java --- I don't know Java. I think at
> least I should have written "(float)1.0" as "1.0f" --- any Java
> programmers who can comment?
>
> -------------------------------------------------------
>
>
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://productguide.itmanagersjournal.com/
> _______________________________________________
> Nutch-developers mailing list
> [EMAIL PROTECTED]
> https://lists.sourceforge.net/lists/listinfo/nutch-developers
>
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
Nutch-developers mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
