Geoff - actually, I did turn of persistent_connections and ignore as well
as set the "head_before_get" setting, and only the head_before_get: true
will result in a good crawl.
> > We have an old web server here that's running "CERN/TSX-32 WWW server
> > version 3.0". The web server only supports HTTP/1.0. Unless I turn
>
> That's OK. Plenty of servers still only support HTTP/1.0.
>
> > on "head_before_get" in my configuration file, it won't even attempt
> > to pull down a single URL from the server.
>
> Try turning off persistent_connections and ignore the head_before_get
> setting:
> <http://www.htdig.org/dev/htdig-3.2/attrs.html#persistent_connections>
>
> My guess is that the HTTP/1.1 persistent connection code isn't properly
> downgrading the connection for the HTTP/1.0 server.
>
> --
> -Geoff Hutchison
___________________________________________________________________________
Visit http://www.visto.com.
Find out how companies are linking mobile users to the
enterprise with Visto.
_______________________________________________
htdig-dev mailing list
[EMAIL PROTECTED]
http://lists.sourceforge.net/lists/listinfo/htdig-dev
