hello, The winter is coming, not much to do outside and I have spent the day working on something I had in mind already for too long.
Unfortunately I couldn't start the implementation as I have thought it could be possible, there are too many nested `select' points in the code and implement an event-driven single-thread parallel wget seems like too much work. I have used different threads, spawned by retrieve_tree in the recur.c, I haven't published the code yet[1] since it is just a ugly hack for now and still sometimes it segfaults, it will take a while before I can go trough the code and ensure it is reentrant and can be used by different threads without problems. But I would like to share some results with you.... $ LANG=C wget --version | head -n 1 GNU Wget 1.13 built on linux-gnu. $ LANG=C ./wget --version | head -n 1 GNU Wget 1.13.4-2567-dirty built on linux-gnu. $ rm -rf it.gnu.org/ && time wget -q --no-http-keep-alive -r -np http://it.gnu.org/~gscrivano/files/parallel/ real 0m2.808s user 0m0.008s sys 0m0.020s $ rm -rf it.gnu.org/ && time ./wget --jobs=2 -q --no-http-keep-alive -r -np http://it.gnu.org/~gscrivano/files/parallel/ real 0m1.291s user 0m0.004s sys 0m0.016s $ rm -rf it.gnu.org/ && time ./wget --jobs=4 -q --no-http-keep-alive -r -np http://it.gnu.org/~gscrivano/files/parallel/ real 0m0.521s user 0m0.008s sys 0m0.012s $ rm -rf it.gnu.org/ && time ./wget --jobs=8 -q --no-http-keep-alive -r -np http://it.gnu.org/~gscrivano/files/parallel/ real 0m0.395s user 0m0.008s sys 0m0.004s Nice eh? :-) Any comment? Suggestion? Insult? Cheers, Giuseppe 1) but the braves can find the current ugly hack here: http://it.gnu.org/~gscrivano/files/parallel_wget.patch
