Re: Parallel downloads provide significant speedup

2006-09-30 Thread Daniel Stenberg 

On Sat, 30 Sep 2006, Anthony L. Bryan wrote:


Multithreaded downloads can increase speed quite a bit.


I don't think anyone has argued about that downloading from several servers at 
once will be faster in many occasions (like when each server gives you less 
bandwidth for the transfer than what you have as download link bandwidth). I 
think what is debatable is wheather to download the same file off the same 
server using several connects.


I believe the HTTP spec says that a "nice" HTTP client connect no more than 
twice to the same server. And twice is then only due to (the effects of) 
pipelining on the first connection and the second would be for getting "out of 
band" data.

Re: Parallel downloads provide significant speedup

2006-09-30 Thread Anthony L. Bryan 
Multithreaded downloads can increase speed quite a bit.

Here's a test from home (6 megabit cable), using aria2 which is similar to
wget but it supports torrents and metalinks. The first example downloads the
file in one segment. The second example downloads the metalink which lists
the mirror locations & checksum of the file, downloads the file in 5
segments from 5 different mirrors, then verifies the checksum - so the
actual download is faster than the time listed.

$ time aria2c
http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.17.10.tar.gz
51,552,272/51,709,946 Bytes 99% 00s 152.41 KB/s 1 connections
The download was complete. <./linux-2.6.17.10.tar.gz>

real5m26.995s
user0m2.040s
sys 0m13.529s

$ time aria2c
http://download.packages.ro/metalink/kernel/linux-2_6_17_10_tar_gz.metalink
0/21,268 Bytes 0% - 0.00 KB/s 1 connections
The download was complete. <./linux-2_6_17_10_tar_gz.metalink>
51,709,946/51,709,946 Bytes 100% 00s 36.77 KB/s 5 connections
The download was complete. <./linux-2.6.17.10.tar.gz>
Now verifying checksum.
This may take some time depending on your PC environment and the size of
file.
checksum OK.

real2m2.496s
user0m1.288s
sys 0m9.145s

(( Anthony Bryan
 )) Metalink [ http://www.metalinker.org ]