This workaround won't solve the generic case, IMHO. As far as I understand (unless I'm missing something obvious), right now, when timestamping is on, any children of a document which is decided to be already mirrored are never mirrored. So, say, if there's a second generation document that is mirrored, but its descendants include some other page to be mirrored, this means that I have to *guess in advance* what branch this is going to occur on to invoke the workaround approach (and to forcibly re-mirror the document which is already mirrored correctly). This renders the whole mirroring logic fundamentally flawed, doesn't it? Mirroring is exactly for PREVENTING of re-download of files which are already downloaded correctly. Current wget only supports this for the "leaf" pages correctly.
The solution IMHO should be simple - in case a file is deemed already mirrored, the download should be replaced by a local read, with all associated parsing and queuing of the children. This is what other mirroring software does (cf. the mm+mirror suite in Perl). I think this should be the default logic with -m; I can't imagine a scenario when "leaf mirroring", the current state of affairs, would be preferred. Vassilii ----- Original Message ----- From: "Herold Heiko" <[EMAIL PROTECTED]> To: "'Vassilii Khachaturov'" <[EMAIL PROTECTED]>; <[EMAIL PROTECTED]> Sent: Tuesday, January 29, 2002 2:52 AM Subject: RE: recursive mirror of new children w/old parent broken (wget-1.8) In other words, you'd want to NOT do timestamping (-N or --timestamping, implicitely turned on by -m) for the first file but do it for later ones. I don't think you can do that currently with one invocation of wget. What you could do is change your script - download (even with -m or -N) that first file only (without recursion, no -r); then start wget a second time with -m -r -i path_to_index.html -B http://newton.uam.mx/xgeorge/ in order to check all those secondary links. Should not be a problem since you do use a script anyway. Heiko > -----Original Message----- > From: Vassilii Khachaturov [mailto:[EMAIL PROTECTED]] > Sent: Tuesday, January 29, 2002 12:11 AM > To: [EMAIL PROTECTED] > Subject: recursive mirror of new children w/old parent broken > (wget-1.8) > > > Hi, > I am using the following script to mirror pages from > http://newton.uam.mx/xgeorge/ > in the local directory (served as http://www.tarunz.org/~xgeorge/): > > wget -q -t10 -c -T120 -nH --cut-dirs=1 -r -m -np > http://newton.uam.mx/xgeorge/ > > Now, if the remote directory index document (mapped to the local > index.html) is not newer than its mirrored local copy, the mirror > process stops; whereas I am expecting it to also check if the > child pages have to be mirrored. This is exhibited by the attached > 'stale.log.gz'. If I forcibly remove the local index.html, > it does mirror index.html, and then proceeds on to properly mirroring > the changed child documents (as seen is the second attachment, > 'forced.log.gz'). > > I hope this is enough to describe the bug; > but I will be happy to answer any additional questions > or test any patch for you. > > Kind regards, > Vassilii
