On 6/17/19 10:32 PM, David Bodin wrote: > *wget --page-requisites --span-hosts --convert-links --adjust-extension > --execute robots=off --user-agent Mozilla > --random-wait > https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/ > <https://www.invisionapp.com/inside-design/essential-steps-designing-empathy/>* > > This command above provides the following stats: > > Total wall clock time: 35s > > Downloaded: 248 files, 39M in 4.2s (9.36 MB/s) > > This website takes about 5 seconds to download and display all files on a > hard refresh in the browser. > > Why is the wall clock time *significantly longer* than the download time > and is there a way to make it faster?
First of all, --random-wait waits 0.5 to 1.5 seconds after each downloaded page. Don't use it - there have been times when web servers blocked fast clients, but that shouldn't be the case today. Wget uses just one connection for downloading, no compression by default, no http/2. You can try Wget2 which uses as many parallel connections as you like, uses compression by default and http/2 if possible. Depending on the HTTP server, Wget2 is often 10x faster then Wget just with it's default settings. You find the latest Wget2 tarball at https://gnuwget.gitlab.io/wget2/wget2-latest.tar.gz. Instructions how to build at https://gitlab.com/gnuwget/wget2/blob/master/README.md Regards, Tim
signature.asc
Description: OpenPGP digital signature