>From: "Ian Abbott" <[EMAIL PROTECTED]> >To: [EMAIL PROTECTED] >CC: "Bruce BrackBill" <[EMAIL PROTECTED]> >Subject: Re: timestamping content-length --ignore-length >Date: Thu, 31 Jan 2002 17:24:11 -0000 > >On 31 Jan 2002 at 8:41, Bruce BrackBill wrote: > > > The problem is, that my web pages are served up by php > > and the content lengh is not defined. So as the manual states > > I use --ignore-length. But when wget retrieves an image > > it slows right down, possibly because it is ignoring > > the content-length. Maybe an option to ignore the > > content length of certain file types ( say text/html ) > > would be an option for upcomming releases of wget. > >The problem is that wget uses persistent connections by default if >the server supports them. As you are using --ignore-length, wget >must wait for more data will arrive while the connection is open. >The persistent connection is closed by the server after a timeout >- as far as it is concerned, it has already completed the request >and is waiting for a new request to re-use the same connection. >This timeout is what is causing the delays you are seeing. > >You can tell wget not to allow persistent connections using the >--no-http-keep-alive option, which should speed things up in your >case. > >By the way, have you tried it without the --ignore-length option >to see if it works?
Thanks for your responce Ian. When I use it without --ignore-length option it appears that wget "SOMETIMES" ignores the last_modified_date OR wget says to itself ( hey, I see the file is older than the local copy, but hey, since the server isn't sending me a content_length i'm just going to download it again anyway :-). According the the manual ( as I read it ) wget should ALWAYS reget the file if it has an empty content length ( even though this is undesirable behavior ). I know, explaing this is very confusing because I see wget not performing as stated in the manual. But I want it to perform incorectly... always ;-) That is, get the file by date and always ignore the last modified date. OK here I go. 1) I want to mirror a local copy of the site. 2) In the php scripts I send out last_modified_date 3) php does not send content_length ( and I don't do it either in the script ) 4) as the manual states wget will look at the last_modified date in a HEAD request and GET the file if it is newer UNLESS content_lenghth is different -this is where the problem lies ( NOT using --ignore-length option ) wget appears to do a head request, see that the file is older and move on . Now SOMETIMES it does this. This is desired behaviour for me of cource, because I don't send a content length for text/html files, and I don't want wget to GET every file EVERY time. But, wget should not do this, ever, because the manual states that content length will override last modifed date. whoa! ;-) I'll try --no-http-keep-alive and see Thanks again, Bruce > >Perhaps the manual ought to mention the undesirability of using >--ignore-length with persistent connections. _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp.
