At 12:03 PM -0400 2002/04/23, Chris Devers wrote:
>On Tue, 23 Apr 2002, Max Horn wrote:
>
>> At 9:43 Uhr -0400 23.04.2002, Chris Zubrzycki wrote:
>> >
>> >How hard would it be to add code to perform x number of downloads at
>> >once, where x is set in the config field? just wondering, for people
>> >who have fast connections.
>>
>> First, you would have to do multiple process (forks). Then you have
>> to manage those somehow.
>
>Basically, re-implement a kinda backwards Apache, except instead of
>serving multiple parallel URLs, you're grabbing them.
>
>Max's points about the complexity of implementing this are all valid. I'll
>just add that, in addition to that complexity/overhead/debugging that this
>would involve, it's also not clear that it would save much time.
>
>Even given that the design issues are thought through & properly
>implemented, I think the best case scenario (assuming that computational
>time of running all this is effectively zero & we're bound instead by
>bandwidth) is that it takes exactly the same amount of time to download
>everything.
That's not true. As you say below, the best case is if your
local pipe can handle all downloads simultaneously, at the max speed
the far-end servers can provide them. You could get 5 quick files,
all simultaneous with getting a single 'slow' file. I know my
download speeds are rarely the speed of my DSL line, but I can often
run 4 simultaneously without visibly affecting download speed.
The ideal implementation would be to use multidownload
capabilities in curl or wget. You'd feed curl a bunch of URLs, and it
would return when they were all fetched, or failed. Unfortunately,
fink's needs are more complicated than they're likely to provide,
since we'd need to provide alternates for each target.
A simple alternative would be for fink to start 6 child
threads, each of which starts curl download. The parent would just
sleep until the children all returned, then restarted at the
beginning of the download process, and discover that all or most of
the packages aren't present. After the first attempt, fink could run
in the current single-download try/fail/retry-alternate mode, getting
human input for which alternatives to try. It's not very elegant, but
this is deliberately a much simpler approach than building all the
intelligence we could take advantage of, in terms of alternatives and
user interaction.
>Think about it: instead of four consequitive downloads that take (making
>up figures here) ten seconds each, you have four simultaneous downloads
>that take forty seconds each, because they're still sharing the same
>constrained bandwidth.
>
>You only stand to gain if this scheme can take advantage of different
>access paths (a second NIC or modem or something) or if the bottleneck is
>the remote server, and not your connection. Sometimes the latter is the
>case -- I think we all seem to be having a slow time getting downloads
>from Sourceforge's site, for example. But in most cases I don't think
>there's going to be enough gain from parallelizing to justify all the work
>it'll take to get it to work reliably.
Chris Pepper
--
Chris Pepper: <http://www.reppep.com/~pepper/>
Rockefeller University: <http://www.rockefeller.edu/>
_______________________________________________
Fink-devel mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/fink-devel
