There is a problem with the way depth-limited recursive retrievals are
performed. This is because the recursion depth tests assume the pages
of the web-site are related to each other in a hierarchical tree,
rather than a directed graph.
To illustrate the problem, here is a diagram:
test.html ---> a.html ---> b.html ---> c.html
--------------->
test.html has a link to a.html and a link to b.html in that order.
a.html has a link to b.html and b.html has a link to c.html.
Now do a level 2 recursive retrieval on test.html :
test.html is retrieved and marked as visited.
+ URLs to a.html and b.html are extracted from test.html.
+ a.html is retrieved, marked as visited and recursed into.
+ + URL to b.html is extracted from a.html.
+ + b.html is retrieved, marked as visited and recursed into.
+ + + recursion depth exceeded!
+ + + (possibly load page requisites here)
+ + + RETURN
+ + Reached end of extracted URL list for a.html
+ + RETURN
+ b.html is skipped because it has already been seen.
+ Reached end of extracted URL list for test.html.
+ RETURN
END
Note that c.html has not been retrieved, even though it can be
considered to be within the range of a level 2 recursive retrieval.
If the order of the URLs in test.html is reversed then c.html would be
retrieved. If the URLs were retrieved in a breadth first fashion
rather than depth first, the problem can be avoided altogether. I'll
have a play around in recur.c to do this - it shouldn't be too hard.
Regards,
Ian Abbott
