Hi,
In my last message I said "[have Wget] get the file by date and
always ignore the last modified date" and I meant to say 'always ignore
the content-lenghth' because I am not sending it. But Ian, you
understood what I meant anyway :-).
And after debugging it appears that Wget is working just fine ( no need
for --ignore-length and the coresponding --no-http-keep-alive )
It IS doing the following:
-Ian wrote-
In that case[ the absence of content-length in the header],
Wget's timestamping retrieval decision is based
solely on the Last-Modified header, regardless of whether you use
--ignore-length or not. A debug log would help confirm this.
I think the manual should state this :-)
Here is a debug log for a page that I reget:
wget.exe -N -l1 -d -UWGET-DEBUG
http://www.myNameBasedVirtualHost.com/thepage.html
DEBUG output created by Wget 1.8.1 on Windows.
--08:21:14-- http://www.myNameBasedVirtualHost.com/thepage.html
=> `thepage.html'
Resolving www.myNameBasedVirtualHost.com... done.
Caching www.myNameBasedVirtualHost.com => server.ip
Connecting to www.myNameBasedVirtualHost.com[server.ip]:80... connected.
Created socket 33.
Releasing 00A008A0 (new refcount 1).
---request begin---
HEAD /thepage.html HTTP/1.0
User-Agent: WGET-DEBUG
Host: www.myNameBasedVirtualHost.com
Accept: */*
Connection: Keep-Alive
---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Sat, 02 Feb 2002 16:21:14 GMT
Server: Apache/1.3.1.1 SSL/1.15 PHP/4.0b2
Last-Modified: Thu, 03 Jan 2002 14:30:22 GMT
Keep-Alive: timeout=45, max=100
Connection: Keep-Alive
Content-Type: text/html
Length: unspecified [text/html]
Closing fd 33
Server file no newer than local file `thepage.html' -- not retrieving.
Regarding Chunked Data:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1
It appears to me that my php/apache is sending out chunked data:
{13} % telnet myRealServer.com 80
Trying real-server-ip...
Connected to myRealServer.com.
Escape character is '^]'.
GET / HTTP/1.1
host:myNameBasedVirtualHost.com
HTTP/1.1 200 OK
Date: Sat, 02 Feb 2002 16:45:43 GMT
Server: Apache/1.3.1.1 SSL/1.15 PHP/4.0b2
Last-Modified: Thu, 03 Jan 2002 14:48:11 GMT
Transfer-Encoding: chunked
Content-Type: text/html
[snip] page contents[/snip]
Thanks,
Bruce
>
>On 1 Feb 2002 at 8:17, Daniel Stenberg wrote:
>
> > On Thu, 31 Jan 2002, Ian Abbott wrote:
> > > The problem is that wget uses persistent connections by default if the
> > > server supports them. As you are using --ignore-length, wget must wait
>for
> > > more data will arrive while the connection is open. The persistent
> > > connection is closed by the server after a timeout - as far as it is
> > > concerned, it has already completed the request and is waiting for a
>new
> > > request to re-use the same connection. This timeout is what is causing
>the
> > > delays you are seeing.
> >
> > If that is what wget does, then yes, it is obstructed by the
>--ignore-length
> > option.
> >
> > The "proper" action (IMHO) would be to use a true HTTP/1.1 request and
>thus
> > most likely receive a chunked transfer-encoded data stream back, which
> > doesn't need a content-length header for the client to figure out when
>its
> > complete.
>
>Does PHP do that?
>
> > > You can tell wget not to allow persistent connections using the
> > > --no-http-keep-alive option, which should speed things up in your
>case.
> >
> > That is not nice, and goes against recommendations in recent HTTP RFCs.
>
>Leaving connections hanging open is not nice, either!
>
> > > By the way, have you tried it without the --ignore-length option to
>see if
> > > it works?
> >
> > Surely it wouldn't be much difference, as the header won't be included
>in the
> > first place.
>
>Yes, you're right of course for the PHP output. But Bruce said it
>slowed right down when he was retrieving images. I assumed those
>would be served by Apache itself without any control by PHP.
>
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp.
