glenn, i know that verisign makes this available for .com and .net as
"TLD zone files".
for ccTLDs like .us and .uk, you'll have to see if the TLD registrar
provides the same. the following page has some useful links to these
folks:
http://www.dnsstuff.com/info/dnslinks.htm
--matt
On Nov 29, 2005, at 10:23 AM, Insurance Squared Inc. wrote:
Along these same lines (as I'm interested in a similiar country-
specific project), is there any place to get a list of all the
domains for a specific TLD to use to seed nutch? i.e. if I wanted
to get a list of all currently registered .it, .de, or .ca's?
I've looked without success. I'm thinking that this information
isn't available due to spamming issues, however in the paper you
referenced they discuss crawling an entire TLD which seemed to
indicate they may have access to this info.
Thanks,
Glenn
Ken Krugler wrote:
Is there anyone that can implement a country crawler? I
estimate around 40m documents. Please send me info about your
prev work and how much time it would take to setup and money :-)
Check out the paper titled "Crawling a Country: Better Strategies
than Breadth-First for Web Page Ordering" by Ricardo Baeza-Yates &
others. They were using a crawl of Chilean domains to test
strategies for efficient crawling, so it seems like it would be of
interest to you.
The main problem we've run into in doing similar limited domain
crawls is that you wind up with many fewer hosts, and thus more
URLs/host in any given fetch loop. The restriction of being polite
(one thread per host) leads to lots of retry errors caused by
fetcher threads blocking on a host (IP address) that is already
being accessed by another fetcher thread, and thus lower pages/
second throughput.
So we've been making some mods to Nutch to improve our
performance, but it's not debugged yet...getting closer, though.
-- Ken
--
Matt Kangas / [EMAIL PROTECTED]
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
Nutch-general mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-general
