On Mon, Apr 03, 2006 at 06:19:51PM -0400, Rob wrote:
> On Mon, Apr 03, 2006 at 05:58:12PM -0400, J. Milgram wrote:
>> But I wonder...
>> Is there a way to develop a list of all domain names?
>> Then you could do:
>> for d in domain_names:
>> if url.open("http://%s"; % d): do stuff
>> elif url.open("http://www.%s"; % d): do stuff
>> elif url.open("http://www.%s/index.html"% d: do stuff
>> etc etc
>> maybe this is faster. Or slower.
> The problem is this looks like port scanning (well, it basically is :-)
> for web addresses. Part of the goal of this project is not to piss
> anyone off... also, how do you come up with the list of domain names?
> Webcrawling is a very established way of walking around lots and lots
> of machines, so no one should raise too much of an eyebrow...
Contact PIR, NetSol, etc., etc. Promise that you won't spam. Pull a list
of second level domains. Hit each one of them, by that name and by www.
Then look for links. :-)
Ben
--
Ben Stern UNIX & Networks Monkey [EMAIL PROTECTED]
This post does not represent FTI, even if I claim it does. Neener neener.
UM Linux Users' Group Electromagnetic Networks Microbrew Software
