-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brock Murch wrote: > I try to keep a mirror of NASA atteph ancilliary data for modis processing. I > know that means little, but I have a cron script that runs 2 times a day. > Sometimes it works, and others, not so much. The sh script is listed at the > end of this email below. As is the contents of the remote ftp server's root > and portions fo the log. > > I don't need all the data on the remote server, only some thus I use > --cut-dirs.To make matters stranger, the software (also from NASA) that uses > these files, looks for them in a single place on the client machine where the > software runs, but needs data from 2 different directories on the remote ftp > server. If the data is not on the client machine, the software kindly ftp's > the files to the local directory. However, I don't allow write access to that > directory as many people use the software and when it is d/l'ed it has the > wrong perms for others to use it, thus I mirror the data I need from the ftp > site locally. In the script below, there are 2 wget commands, but they are to > slightly different directories (MODISA & MODIST).
I wouldn't recommend that. Using the same output directory for two different source directories seems likely to lead to problems. You'd most likely be better off by pulling to two locations, and then combining them afterwards. I don't know for sure that it _will_ cause problems (except if they happen to have same-named files), as long as .listing files are being properly removed (there were some recently-fixed bugs related to that, I think? ...just appending new listings on top of existing files). > It appears to me that the problem occurs if there is a ftp server error, and > wget starts a retry. wget goes to the server root, gets the .listing from > there for some reason (as opposed to the directory it should go to on the > server), and then goes to the dir it needs to mirror and can't find the files > (that are listed in the root dir) and creates dirs, and then I get "No such > file" errors and recursive directories created. Any advice would be > appreciated. This snippet seems to be the source of the problem: > Error in server response, closing control connection. > Retrying. > > - --14:53:53-- ftp://oceans.gsfc.nasa.gov/MODIST/ATTEPH/2002/110/ > (try: 2) => `/home1/software/modis/atteph/2002/110/.listing' > Connecting to oceans.gsfc.nasa.gov|169.154.128.45|:21... connected. > Logging in as anonymous ... Logged in! > ==> SYST ... done. ==> PWD ... done. > ==> TYPE I ... done. ==> CWD not required. > ==> PASV ... done. ==> LIST ... done. That "CWD not required" bit is erroneous. I'm 90% sure we fixed this issue recently (though I'm not 100% sure that it went to release: I believe so). I believe we made some related fixes more recently. You provided a great amount of useful information, but one thing that seems to be missing (or I missed it) is the Wget version number. Judging from the log, I'd say it's 1.10.2 or older; the most recent version of Wget is 1.11.4; could you please try to verify whether Wget continues to exhibit this problem in the latest release version? I'll also try to look into this as I have time (but it might be awhile before I can give it some serious attention; it'd be very helpful if you could do a little more legwork). - -- Thanks very much, Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJBgNh7M8hyUobTrERAuGoAKCCUoBN0sURKA/51x0o4HN59K8+AACfUYuj i8XW58MvjvbS3oy4OsOmbpc= =4kpD -----END PGP SIGNATURE-----
