Hash: SHA1

Brock Murch wrote:
> I try to keep a mirror of NASA atteph ancilliary data for modis processing. I 
> know that means little, but I have a cron script that runs 2 times a day. 
> Sometimes it works, and others, not so much. The sh script is listed at the 
> end of this email below. As is the contents of the remote ftp server's root 
> and portions fo the log. 
> I don't need all the data on the remote server, only some thus I use 
> --cut-dirs.To make matters stranger, the software (also from NASA) that uses 
> these files, looks for them in a single place on the client machine where the 
> software runs, but needs data from 2 different directories on the remote ftp 
> server. If the data is not on the client machine, the software kindly ftp's 
> the files to the local directory. However, I don't allow write access to that 
> directory as many people use the software and when it is d/l'ed it has the 
> wrong perms for others to use it, thus I mirror the data I need from the ftp 
> site locally. In the script below, there are 2 wget commands, but they are to 
> slightly different directories (MODISA & MODIST).

I wouldn't recommend that. Using the same output directory for two
different source directories seems likely to lead to problems. You'd
most likely be better off by pulling to two locations, and then
combining them afterwards.

I don't know for sure that it _will_ cause problems (except if they
happen to have same-named files), as long as .listing files are being
properly removed (there were some recently-fixed bugs related to that, I
think? ...just appending new listings on top of existing files).

> It appears to me that the problem occurs if there is a ftp server error, and 
> wget starts a retry. wget goes to the server root, gets the .listing from 
> there for some reason (as opposed to the directory it should go to on the 
> server), and then goes to the dir it needs to mirror and can't find the files 
> (that are listed in the root dir) and creates dirs, and then I get "No such 
> file" errors and recursive directories created. Any advice would be 
> appreciated.

This snippet seems to be the source of the problem:

> Error in server response, closing control connection.
> Retrying.
> - --14:53:53--  ftp://oceans.gsfc.nasa.gov/MODIST/ATTEPH/2002/110/
>   (try: 2) => `/home1/software/modis/atteph/2002/110/.listing'
> Connecting to oceans.gsfc.nasa.gov||:21... connected.
> Logging in as anonymous ... Logged in!
> ==> SYST ... done.    ==> PWD ... done.
> ==> TYPE I ... done.  ==> CWD not required.
> ==> PASV ... done.    ==> LIST ... done.

That "CWD not required" bit is erroneous. I'm 90% sure we fixed this
issue recently (though I'm not 100% sure that it went to release: I
believe so).

I believe we made some related fixes more recently. You provided a great
amount of useful information, but one thing that seems to be missing (or
I missed it) is the Wget version number. Judging from the log, I'd say
it's 1.10.2 or older; the most recent version of Wget is 1.11.4; could
you please try to verify whether Wget continues to exhibit this problem
in the latest release version?

I'll also try to look into this as I have time (but it might be awhile
before I can give it some serious attention; it'd be very helpful if you
could do a little more legwork).

- --
Thanks very much,
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


Reply via email to