Hash: SHA1

Okay, so there's been a lot of thought in the past, regarding better
extensibility features for Wget. Things like hooks for adding support
for traversal of new Content-Types besides text/html, or adding some
form of JavaScript support, or support for MetaLink. Also, support for
being able to filter results pre- and post-processing by Wget: for
example, being able to do some filtering on the HTML to change how Wget
sees it before parsing for links, but without affecting the actual
downloaded version; or filtering the links themselves to alter what Wget

The original concept before I came onboard, was plugin modules. After
some thought, I'd decided I didn't like this overly much, and have
mainly been leading toward the idea of a next-gen Wget-as-a-library
thing, probably wrapping libcurl (and with a command-client version,
like curl). This obviously wouldn't have been a Wget any more, so would
have been a separate project, with a different name.

However, another thing that's been vaguely itching at me lately, is the
fact that Wget's design is not particularly unix-y. Instead of doing one
thing, and doing it well, it does a lot of things, some well, some not.

So the last couple days I've been thinking, maybe "wget-ng" should be a
suite of interoperating shell utilities, rather than a library or a
single app. This could have some really huge advantages: users could
choose their own html-parser to use, they can plug in parsers for
whatever filetypes they desire, people who want to implement exotic
features can do that...

Of course, at this point we're talking about something that's
fundamentally different from "Wget". Just as we were when we were
considering making a next-gen library version. It'd be a completely
separate project. And I'm still not going to start it right away (though
I think some preliminary requirements and design discussions would be a
good idea). Wget's not going to die, nor is everyone going to want to
switch to some new-fangled re-envisioning of it.

But the thing everyone loves about Unix and GNU (and certainly the thing
that drew me to them), is the bunch-of-tools-on-a-crazy-pipeline
paradigm, which is what enables you to mix-and-match different tools to
cover the different areas of functionality. Wget doesn't fit very well
into that scheme, and I think it could become even much more powerful
than it already is, by being broken into smaller, more discreet,
projects. Or, to be more precise, to offer an alternative that does the

So far, the following principles have struck me as advisable for a
project such as this:

 - The tools themselves, as much as possible, should be written in an
easily-hackable scripting language. Python makes a good candidate. Where
we want efficiency, we can implement modules in C to do the work.

 - While efficiency won't be the highest priority (else we'd just stick
to the monolith), it's still important. Spawning off separate processes
to each fetch their own page, initiating a new connection each time,
would be a lousy idea. So, the architectural model should center around
a "URL-getter" driver, that manages connections and such, reusing
persistent ones as much as possible. Of course, there might be distinct
commands to handle separate types of URLs, (or alternative methods for
handling them, such as MetaLink), and perhaps not all of these would be
able to do persistence (a dead-simple way to add support for scp, etc,
might be to simply call the command-line program).

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org


Reply via email to