This is for the Nutch crew,
I was reading in the paper this morning that you (the Nutch group) were looking to build a 1 billion URL database, and while I only have some 10 million URLS, I will happily share my crawl if you are still wanting to do this project. Admitted it is just 1% of what you are looking to build, but if you want it, I can zip it, dump it on a USB drive, and ship it to you. More than happy to do that. My crawl is mostly centered on the pacific rim URLS, china, Japan, government web sites, and other places like that. If you are interested, give me a shout, and I'll have that puppy in the mail, and will send regular updates as they become available. Cheers/r/dan
