Hi, I'm using wget to recursively retrieve a directory tree from an FTP server. Occasionally there are connection failures (thanks to a flaky server, or possibly dodgy network infrastructure on the route, but that's not the problem I'm concerned with here), so I'm relying on wget to retry failed downloads. I'm seeing a problem where the retry "succeeds" but results in an empty file.
I've included log output (using the -d switch) from when this happens below. You'll see that for the retry wget sends a REST command to reset the start position before starting the RETR command. I'm confused about the argument to REST: 51132. It's the full length in bytes of the file to be retrieved. The RETR then shows the entire contents of the file being skipped, and wget announces that it successfully retrieved and saved 0 bytes. Is that argument to REST correct? $ wget --version GNU Wget 1.11.4 $ wget ftp://ftp.site.com/Dir/All/dirname --ftp-user=user --ftp-password=password --recursive --no-host-directories --no-parent --cut-dirs=2 --timestamping --output-file output.log -P target-dir -d log output: ... skipped successful stuff ... --2009-06-10 09:50:54-- ftp://ftp.site.com/Dir/All/dirname/subdir/dirname => `target-dir/dirname/subdir/.listing' changing working directory Prepended initial PWD to relative path: pwd: '/' old: 'Dir/All/dirname/subdir' new: '/Dir/All/dirname/subdir' ==> CWD /Dir/All/dirname/subdir ... --> CWD /Dir/All/dirname/subdir 250 Directory successfully changed. done. conaddr is: XXX.XXX.XXX.XXX ==> PASV ... --> PASV 227 Entering Passive Mode (XXX,XXX,XXX,XX,XXX,XXX) trying to connect to XXX.XXX.XXX.XXX port 32175 Created socket 5. done. ==> LIST ... --> LIST -a 150 Here comes the directory listing. done. 0K 82.0K=0.002s Closed fd 5 226 Directory send OK. 2009-06-10 09:50:54 (82.0 KB/s) - `target-dir/dirname/subdir/.listing' saved [194] DIRECTORY; perms 775; size: 2048; month: Jun; day: 9; time: 15:12:00 (no yr); Ignoring `.' and `..'; Skipping. DIRECTORY; perms 775; size: 118784; month: Jun; day: 9; time: 23:18:00 (no yr); Ignoring `.' and `..'; Skipping. PLAINFILE; perms 664; size: 51132; month: Jun; day: 9; time: 15:12:00 (no yr); filename.txt Removed `target-dir/dirname/subdir/.listing'. --2009-06-10 09:50:54-- ftp://ftp.site.com/Dir/All/dirname/subdir /filename.txt => `target-dir/dirname/subdir/filename.txt' ==> CWD not required. conaddr is: XXX.XXX.XXX.XXX ==> PASV ... --> PASV 27 Entering Passive Mode (XXX,XXX,XX,XX,XX,XX) 421 Timeout. Cannot initiate PASV transfer. conaddr is: XXX.XXX.XXX.XXX ==> PORT ... conaddr is: XXX.XXX.XXX.XXX Local socket fd 5 bound. binding to address XXX.XXX.XXX.XXX using port 40152. --> PORT XX,XX,XXX,XX,XXX,XXX Closed fd 5 Error in server response, closing control connection. Closed fd 4 Closed fd 5 Retrying. --2009-06-10 10:00:55-- ftp://ftp.site.com/Dir/All/dirname/subdir/filename.txt (try: 2) => `target-dir/dirname/subdir/filename.txt' Found ftp.site.com in host_name_addresses_map (0x5567b0) Connecting to ftp.site.com|XXX.XXX.XXX.XXX|:21... connected. Created socket 4. Releasing 0x00000000005567b0 (new refcount 1). Logging in as user ... 220 Welcome to Site FTP service. --> USER user 331 Please specify the password. --> PASS password 230 Login successful. Logged in! ==> SYST ... --> SYST 215 UNIX Type: L8 done. ==> PWD ... --> PWD 257 "/" done. ==> TYPE I ... --> TYPE I 200 Switching to Binary mode. done. changing working directory Prepended initial PWD to relative path: pwd: '/' old: 'Dir/All/dirname/subdir' new: '/Dir/All/dirname/subdir' ==> CWD /Dir/All/dirname/subdir ... --> CWD /Dir/All/dirname/subdir 250 Directory successfully changed. done. conaddr is: XXX.XXX.XXX.XXX ==> PASV ... --> PASV 227 Entering Passive Mode (XXX,XXX,XXX,XX,XXX,XXX) trying to connect to XXX.XXX.XXX.XXX port 31681 Created socket 5. done. ==> REST 51132 ... --> REST 51132 350 Restart position accepted (51132). done. ==> RETR filename.txt ... --> RETR filename.txt 150 Opening BINARY mode data connection for filename.txt (51132 bytes). done. Length: 51132 (50K), 0 (0) remaining 0K ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,, 100% 0.00 =0s Closed fd 5 226 File send OK. 2009-06-10 10:00:55 (0.00 B/s) - `target-dir/dirname/subdir/filename.txt' saved [51132] ... skipped more successful stuff ...
