> This is a good idea in any case. I'll put it in as an option.
Mike and I have both suggested this at different times about a year
ago or more. A simple socket callback should be all that is required.
Additionally, and something that perl has in the LWP modules, is the ability
to test for HEAD first, sticking successful URLs into an array of @VALIDURL,
and then ones which fail the HEAD test, stick in @BADURL, at the end of the
parse, run through @BADURL and see if they are all still bad, and then
ignore them in the gather process. Yes, it's a bit of an architecture change
to the way the present Python code works, but should speed up things
considerably. Test for HEAD, then do your GET.
/d
