On Mon, 3 Apr 2006, Rob wrote:
wget does https, at least the one I have (v. 1.9.1)
Shows what I know :-)
you probably have 1.9.0 :)
The problem is this looks like port scanning (well, it basically is :-)
for web addresses. Part of the goal of this project is not to piss
anyone off... also, how do you come up with the list of domain names?
There are some sites that offer tools for searching availability of domain
names. They may do the trick, maybe. Try googling "list of all domain
names", that led me to a couple. Doesn't solve the pissing people off
problem though.
Webcrawling is a very established way of walking around lots and lots
of machines, so no one should raise too much of an eyebrow...
Leads to the interesting question of whether you can
actually reach every web site from the first one, or is the graph not
completely connected (or however it is one says that).
