URL:
<http://savannah.gnu.org/bugs/?33825>
Summary: 'wget -nc' race condition
Project: GNU Wget
Submitted by: nok
Submitted on: Sa 23 Jul 2011 10:42:56 CEST
Category: Program Logic
Severity: 3 - Normal
Priority: 5 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Name:
Originator Email:
Open/Closed: Open
Discussion Lock: Any
Release: 1.12
Operating System: GNU/Linux
Reproducibility: None
Fixed Release: None
Planned Release: None
Regression: None
Work Required: None
Patch Included: None
_______________________________________________________
Details:
Hello,
I don't know if you aggree with the described problem but I forward it anyway
to point you to this possible problem:
http://bugs.debian.org/500395
--<8--
I downloaded all the aircraft for flightgear from their ftp mirrors. To spread
the
load over several mirrors I did:
wget -nc ftp://ftp.kingmont.com/flightsims/flightgear/Aircraft/*
wget -nc ftp://ftp.de.flightgear.org/pub/fgfs/Aircraft/*
wget -nc ftp://some.third.mirror/pub/fgfs/Aircraft/*
It worked basically as files started by one wget were not downloaded by other
wgets. However, after one wget decided to download a locally non existant file
it
appearently created it only when the data started coming. In this interval -
from
decision to file creation - other wgets sometimes would also decide to
download
the same file and so two (or possibly three) wgets would download and write
the
same file and waste time and bandwidth.
Maybe the destination file should be created right after wget decides to
download
it. Maybe these operations can be combined atomically to avoid the race
completely.
If it later turns out that the file cannot be downloaded, it can be deleted.
That might be a new race as other wegts won't try to download a file which one
wget fails to download.
Bernhard
--8<--
_______________________________________________________
Reply to this item at:
<http://savannah.gnu.org/bugs/?33825>
_______________________________________________
Nachricht geschickt von/durch Savannah
http://savannah.gnu.org/
--
To UNSUBSCRIBE, email to [email protected]
with a subject of "unsubscribe". Trouble? Contact [email protected]
