On Tue, Jan 29, 2002 at 04:54:17PM +0100, Andre Majorel wrote:
>On 2002-01-29 09:56 -0500, Alan Eldridge wrote:
>
>> In particular, does wget parse and follow links in an HTML document,
>> when that document is retrieved (using -r) via the FTP protocol? If
>> not, why not?
>
>I'm inclined to think that recursive retrieval without parsing
>is a feature. HTML content is normally served over HTTP. If you
>want to retrieve HTML through FTP, it's likely because you do
>*not* want to follow the links.
I agree with you. I'm not making a case for doing it, but merely
bringing up that there are two different recursive retrieval models,
one of which (FTP) is (IIRC) a true mirror.
One of the properties of "mirroring" is that links are followed all
the way down the tree; that is, the tree is not "pruned" (depth-wise)
just because a node at a given level has not been changed.
It seems to me that there are some abstractions that can be made about
the retrieval process.
In particular, there's:
(1) the "continuation protocol": does the retrieval continue
to futher depth once an unmodified node is encountered?
(2) the "child-list acquisition protocol": do we get the list
of children of this node by (a) examining metadata or (b) parsing
the contents of the node as a document?
IOW, recursive retrieval can be thought of as a generalized process,
parameterized by protocols that determine that actions and state
transitions at each node in the tree. Ideally, the code that implements
recursive retrieval would know nothing about either the communication
protocol (ftp vs. http) or the data/metadata formats (having to do a special
operation - an ftp listing - on a "directory" node for ftp, vs having to
do a special operation - an html parse - on a particular type of file
node for an http traversal).
--
Alan E
"Please rush me my portable walrus cleaning kit! Yes I am over 18,
but my IQ isn't."
