On Sun, 18 Dec 2011, Alex wrote:
Thanks for your suggestions. I don't think there is any problem, just an inefficiency. I mean, every single page's speed might be limited by the server. Besides, pages are small (160 KB on average, based on 70 000 pages crawled so far), so overhead of resolving DNS names, creating thread, opening socket and so on is significant.
Yes, you're right. If you want to optimize for some kind of transfer performance you should probably consider what to do to minimize the impact of all those details. Such as perhaps not dynamically start new threads or even use less threads.
Does curl tool implement any kind of parallel download?
Nope. It may one day since libcurl provides an API for it, but it's not being worked on.
I currently use "easy" interface, because it really is very easy to use. I took a look at "multi" documentation, but didn't understand how to use it. Besides, I really like to manage threads myself, so that I understand what's going on inside my program. Will multi interface be more efficient than my naive approach?
If you're aiming at a very large amount of parallel transfers, if you rather want an non-blocking API etc then the multi interface is your friend.
We basically provide a bunch of different ways to do transfers, but you get to pick which to use yourself!
-- / daniel.haxx.se ------------------------------------------------------------------- List admin: http://cool.haxx.se/list/listinfo/curl-library Etiquette: http://curl.haxx.se/mail/etiquette.html
