Hash: SHA256

With talk of supporting multiple simultaneous connections in a
next-generation version of Wget, various things have been tumbling
around in my mind.

First off is that I would not wish to do such a thing with threads.
Threads introduce too many problems of their own, including portability
and debugability. I'd much prefer to do asynchronous I/O.

With the use of asynchronous I/O, a (possibly) better way to do
- --timeout presents itself: we can do the appropriate timeouts in our
calls to select(). The main advantage to this is that we don't have to
muck around with signals, signal handling, various portability issues,
etc. We can do one --timeout and be done.

The primary downside to this is that potentially blocking, not directly
I/O things don't get timed out anymore. The only thing that currently
comes to mind is gethostbyname(), which obviously can block, but can't
be select()ed or set to some sort of non-blocking mode. Also, even aside
from --timeout, having all other traffic sit around and wait until a
name is resolved is not really desirable.

The obvious solution to that is to use c-ares, which does exactly that:
handle DNS queries asynchronously. Actually, I didn't know this until
just now, but c-ares was split off from ares to meet the needs of the
curl developers. :)

Of course, if we're doing asynchronous net I/O stuff, rather than
reinvent the wheel and try to maintain portability for new stuff, we're
better off using a prepackaged deal, if one exists. Luckily, one does; a
friend of mine (William Ahern) wrote a package called libevnet that
handles all of that; it wraps libevent (by Niels Provos, for handling
async I/O very portably and using the best available interfaces on the
given system) with higher-level socket and buffer I/O facilities and,
and provides a wrapper around c-ares that makes it convenient to use
with liblookup. If we're going to do async I/O, using libevent and
c-ares, or something very like them, is far too convenient not to do,
and after that decision is made, libevnet becomes a clear win too.

So, the obvious win is that using libevnet, libevent and c-ares gives us
 a "shortest path" to using async I/O, having multiple simultaneous
connections and async DNS queries, and a potentially better way to
manage timeouts.

The obvious loss, and one which I'm positive many of you are already
screaming at me about, is that we just added 3 library dependencies to
Wget in one go. Not freaking cool. Not freaking cool AT ALL.

- -= Wget's Strongest Points =-

I absolutely do not want to require a bunch of libraries in order for
people to build Wget. AFAICT, the vast majority of Wget's user base,
which is probably system packagers and distributors, use it for just the
following reasons:

  1. It's pretty small. Only dependency is OpenSSL, which isn't even
required, but of course in general nobody really doesn't want SSL. (Ooh
looky! Double negatives!)
  2. It's robust. Connection dropped? No prob, try again.
  3. It avoids mucking with preexisting files. Downloading a file named
"foo", but you already _have_ a "foo"? No prob, let's call it "foo.1".

To my mind, these are the core values that have led to so many different
distributions and large software packages relying on Wget. Messing with
any one of these is likely to lose Wget "customers", and in our largest
"target market". (DISCLAIMER: naturally I have nothing whatsoever to
back these claims up. It's conjecture. But it seems pretty credible to me.)

Another major "market" for Wget is the typical command-line "power
user", who uses Wget not only to grab off a quick file, but also to grab
whole sections of sites recursively, and perhaps with occasional quirky
needs like only-visit-these-domains or only-download-these-file-types.
For these people, while point #1 above probably holds relatively little
value, probably being replaced primarily by Wget's HTML-crawling
functionality. In addition to these, points that I believe are highly
desirable to such users are:

  - Being able to tell Wget precisely which files to download and which
to skip. The more expressive power we have to accomplish this the
better. Wget already has remarkable flexibility in this area; but there
are many more things that are desirable, and some of the existing
interface is not up to the task of really powerful expression in this area.
  - Being able to parse and "recursively descend" CSS is really, really
  - Being able to do multiple connections, potentially accelerating the
total download time (mainly for multi-host sessions), would be a win.
  - Being able to extend Wget, to grok new filetypes for recursive
descent (such as non-HTML XML files, or JavaScript), or extend the power
of expression of "what to grab" even further.

- -= The Two Wgets =-

It seems to me, then, that what's really required may in fact be two
different "Wgets".

One that is lightweight but packs a punch: basically Wget as it already
currently is. Making it DTRT where it doesn't, such as with its
expectations of FTP servers, or how it handles HTTP authentication; and
adding CSS support would be _really_ helpful. In order to _keep_ it
lightweight, it would be necessary to keep a tight throttle on what new
functionality is accepted; it would be primarily _maintained_, and not
_developed_, though it would of course be kept up-to-date with evolving
definitions of what the World Wide Web is (CSS being an excellent
example). It would support recursive web fetching, but wouldn't bend
over backwards to handle the more exotic needs.

The other Wget would get all the "cool" stuff: pretty much everything
that has been planned for the "next-gen", "2.0" version of Wget. Its
focus would be on users that want it to be their "everything" tool, and
damn the hard-disk-space requirements and library dependencies (not
getting _too_ crazy, of course--that's what the plugin architecture is

This would certainly allay a growing fear I've had, that a lot of what
people were getting excited about in discussions of "Wget 2.0", just
plain doesn't _feel_ right for Wget. But were unquestionably useful
feature additions. I initially quelled this concern with the thought
that I could simply sequester the really exotic features to plugins,
allowing people the freedom to choose what they want their Wget to be.

But asynchronous I/O can't be simply partitioned away like that: it
requires intrinsic and pervasive changes to Wget's architecture. While
the two Wgets could share some logic for recursion, file naming,
timestamping, etc, the actual I/O wouldn't easily be sharable. Well:
code written for an async I/O platform can easily just be used
synchronously, but async code comes at a significant cost to legibility
and flexibility, that IMO woudn't be worth the cost in the "synchronous"

Plus, there is the following thought. While I've talked about not
reinventing the wheel, using existing packages to save us the trouble of
having to maintain portable async code, higher-level buffered-IO and
network comm code, etc, I've been neglecting one more package choice.
There is, after all, already a Free Software package that goes beyond
handling asynchronous network operations, to specifically handle
asynchronous _web_ operations; I'm speaking, of course, of libcurl.
There would seem to be some obvious motivation for simply using libcurl
to handle all asynchronous web traffic, and wrapping it with the logic
we need to handle retries, recursion, timestamping, traversing,
selecting which files to download, etc. Besides async web code, of
course, we'd also automatically get support for a number of various
protocols (SFTP, for example) that have been requested in Wget.

PLEASE NOTE: these are ramblings. They are ideas. They are what's
currently rattling around in my brain. Note too, that there's a couple
leaps in the given logic for having a completely separate "Wget 2.0",
the biggest one probably being that multiple connections does not
automatically implying asynchronous I/O; that's just my preference.

Not going for async I/O destroys the depends-on-myriad-libraries
argument, and the whole Wget-2.0-needs-to-be-separate, and
maybe-we-should-use-libcurl arguments. OTOH, the other options I can
think of--using threads, or using multiple processes--have their own
strong downsides, especially in terms of portability and maintenance cost.

I expect this to be controversial thinking, and am hereby officially
begging for feedback, and for alternative thoughts and viewpoints.

And, of course, when I say "there would be two Wgets", what I really
mean by that is that the more exotic-featured one would be something
else entirely than a Wget, and would have a separate name.

- --
Micah J. Cowan
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


Reply via email to