On Fri, 26 Oct 2007, Micah Cowan wrote:
The obvious solution to that is to use c-ares, which does exactly that:
handle DNS queries asynchronously. Actually, I didn't know this until just
now, but c-ares was split off from ares to meet the needs of the curl
developers. :)
We needed an asynch name resolver for libcurl so c-ares started out that way,
but perhaps mostly because the original author didn't care much for our
improvements and bug fixes. ADNS is a known alternative, but we couldn't use
that due to license restrictions. You (wget) don't have that same problem with
it. I'm not able to compare them though, as I never used ADNS...
Of course, if we're doing asynchronous net I/O stuff, rather than reinvent
the wheel and try to maintain portability for new stuff, we're better off
using a prepackaged deal, if one exists. Luckily, one does; a friend of mine
(William Ahern) wrote a package called libevnet that handles all of that;
When I made libcurl grok a vast number of simultaneous connections, I went
straight with libevent for my test and example code. It's solid and fairly
easy to use... Perhaps libevnet makes it even easier, I don't know.
Plus, there is the following thought. While I've talked about not
reinventing the wheel, using existing packages to save us the trouble of
having to maintain portable async code, higher-level buffered-IO and network
comm code, etc, I've been neglecting one more package choice. There is,
after all, already a Free Software package that goes beyond handling
asynchronous network operations, to specifically handle asynchronous _web_
operations; I'm speaking, of course, of libcurl.
I guess I'm not the man to ask nor comment this a lot, but look what I found:
http://www.mail-archive.com/wget@sunsite.dk/msg01129.html
I've always thought and I still believe that wget's power and most appreciated
abilities are in the features it adds on top of the transfer, like HTML
parsing, ftp list parsing and the other things you mentioned.
Of course, going one single unified transfer library is perhaps not the best
thing from a software eco-system perspective, as competition tends to drive
innovation and development, but the more users of a free software/open source
project we get the better it will become.
