On Fri, 26 Oct 2007, Micah Cowan wrote:

The obvious solution to that is to use c-ares, which does exactly that: handle DNS queries asynchronously. Actually, I didn't know this until just now, but c-ares was split off from ares to meet the needs of the curl developers. :)

We needed an asynch name resolver for libcurl so c-ares started out that way, but perhaps mostly because the original author didn't care much for our improvements and bug fixes. ADNS is a known alternative, but we couldn't use that due to license restrictions. You (wget) don't have that same problem with it. I'm not able to compare them though, as I never used ADNS...

Of course, if we're doing asynchronous net I/O stuff, rather than reinvent the wheel and try to maintain portability for new stuff, we're better off using a prepackaged deal, if one exists. Luckily, one does; a friend of mine (William Ahern) wrote a package called libevnet that handles all of that;

When I made libcurl grok a vast number of simultaneous connections, I went straight with libevent for my test and example code. It's solid and fairly easy to use... Perhaps libevnet makes it even easier, I don't know.

Plus, there is the following thought. While I've talked about not reinventing the wheel, using existing packages to save us the trouble of having to maintain portable async code, higher-level buffered-IO and network comm code, etc, I've been neglecting one more package choice. There is, after all, already a Free Software package that goes beyond handling asynchronous network operations, to specifically handle asynchronous _web_ operations; I'm speaking, of course, of libcurl.

I guess I'm not the man to ask nor comment this a lot, but look what I found:


I've always thought and I still believe that wget's power and most appreciated abilities are in the features it adds on top of the transfer, like HTML parsing, ftp list parsing and the other things you mentioned.

Of course, going one single unified transfer library is perhaps not the best thing from a software eco-system perspective, as competition tends to drive innovation and development, but the more users of a free software/open source project we get the better it will become.

Reply via email to