Evil Twin <[EMAIL PROTECTED]> writes:
> I just discovered wget while looking for a solution to an upcoming
> problem. I was hoping someone on the list might be able to help me with
> configuring wget. Here's the problem.
>
> I have a ship at sea that will have 24/7 network connectivity via
> C-band satellite comm. The bandwidth is 2Mb/sec (I am not sure how
> reliable the C-band comm will be). The ship is collecting real-time
> bathymetric/hydrographic/oceanographic data at a much lower rate (for
> the data in which I am interested). Data files are appended to until
> the file names are changed, generally at the end of a survey line. I
> would like to mirror some of the collected data to a Linux system
> inhouse but, for those files that are open and being appended to, I
> don't want to resend the entire file, just the portion that I don't
> have. So, the directories that I want to mirror will consist of closed
> files and files that are being appended to. From reading the
> documentation for wget it appears that it is possible to do this. I was
> thinking in terms of checking for new data every 10-15 minutes. I have
> root privileges on all the machines on the ship and inhouse. Can anyone
> help me with configuring wget to do this?
Basically you just need to use --timestamping to make sure you don't re-get
unchanged files you already have, and --continue to grab just the part
appended to the end of your datafiles.
Note that --continue doesn't have any smarts -- if you use it for an entire
recursive download, any file that the server provides a timestamp for
(e.g. not CGI output, usually) will be treated as a potentially
partially-downloaded file. So if you have some file called "readme.html"
and you make it bigger at some point, --continue will only download the
extra bytes at the end of the file, which will be wrong unless the change to
the file really was nothing but an append, and no changes earlier in the
file.
If that's a problem you'll have to do multiple runs of wget, only using
--continue where appropriate (e.g. only in certain directories).
Right now the documentation doesn't say anything about using --continue in
this manner, nor does it mention anything about using it in conjunction with
--recursive. I'll fix that right now.
---------------------------------------------------------------
Dan Harkless | To help prevent SPAM contamination,
GNU Wget co-maintainer | please do not mention this email
http://sunsite.dk/wget/ | address in Usenet posts -- thank you.
